From jychang48 at gmail.com  Fri Feb  1 01:38:34 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 1 Feb 2019 00:38:34 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
Message-ID: <CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>

So my matrix doesn't quite look like that, since I am solving a three-phase
unbalanced system. Here's an example of what my matrix looks like for the
IEEE-4 bus system with three-phases (so 12 total dofs):

Mat Object: 1 MPI processes
  type: seqaij
row 0: (0, 344.553 - 1240.43 i) (1, 30.7034 + 10.9125 i) (2, 31.1021 +
10.5924 i) (3, -1.28446 + 2.66353 i) (4, 0.623954 - 0.914098 i) (5,
0.225182 - 0.593995 i)
row 1: (0, 30.7034 + 10.9125 i) (1, 344.705 - 1240.53 i) (2, 30.945 +
10.7235 i) (3, 0.623954 - 0.914098 i) (4, -1.43666 + 2.76012 i) (5,
0.382357 - 0.725045 i)
row 2: (0, 31.1021 + 10.5924 i) (1, 30.945 + 10.7235 i) (2, 344.423 -
1240.34 i) (3, 0.225182 - 0.593995 i) (4, 0.382357 - 0.725045 i) (5,
-1.15424 + 2.57087 i)
row 3: (0, -1.28446 + 2.66353 i) (1, 0.623954 - 0.914098 i) (2, 0.225182 -
0.593995 i) (3, 1.38874 - 3.28923 i) (4, -0.623954 + 0.914098 i) (5,
-0.225182 + 0.593995 i) (6, -0.312601 + 1.8756 i)
row 4: (0, 0.623954 - 0.914098 i) (1, -1.43666 + 2.76012 i) (2, 0.382357 -
0.725045 i) (3, -0.623954 + 0.914098 i) (4, 1.54094 - 3.38582 i) (5,
-0.382357 + 0.725045 i) (7, -0.312601 + 1.8756 i)
row 5: (0, 0.225182 - 0.593995 i) (1, 0.382357 - 0.725045 i) (2, -1.15424 +
2.57087 i) (3, -0.225182 + 0.593995 i) (4, -0.382357 + 0.725045 i) (5,
1.25852 - 3.19657 i) (8, -0.312601 + 1.8756 i)
row 6: (3, -0.312601 + 1.8756 i) (6, 1.96462 - 7.75312 i) (7, -0.499163 +
0.731278 i) (8, -0.180145 + 0.475196 i) (9, -1.02757 + 2.13082 i) (10,
0.499163 - 0.731279 i) (11, 0.180145 - 0.475196 i)
row 7: (4, -0.312601 + 1.8756 i) (6, -0.499163 + 0.731278 i) (7, 2.08637 -
7.8304 i) (8, -0.305885 + 0.580036 i) (9, 0.499163 - 0.731279 i) (10,
-1.14932 + 2.2081 i) (11, 0.305885 - 0.580036 i)
row 8: (5, -0.312601 + 1.8756 i) (6, -0.180145 + 0.475196 i) (7, -0.305885
+ 0.580036 i) (8, 1.86044 - 7.679 i) (9, 0.180145 - 0.475196 i) (10,
0.305885 - 0.580036 i) (11, -0.923391 + 2.0567 i)
row 9: (6, -1.02757 + 2.13082 i) (7, 0.499163 - 0.731279 i) (8, 0.180145 -
0.475196 i) (9, 1.3396 - 2.28195 i) (10, -0.499163 + 0.731278 i) (11,
-0.180145 + 0.475196 i)
row 10: (6, 0.499163 - 0.731279 i) (7, -1.14932 + 2.2081 i) (8, 0.305885 -
0.580036 i) (9, -0.499163 + 0.731278 i) (10, 1.46136 - 2.35922 i) (11,
-0.305885 + 0.580036 i)
row 11: (6, 0.180145 - 0.475196 i) (7, 0.305885 - 0.580036 i) (8, -0.923391
+ 2.0567 i) (9, -0.180145 + 0.475196 i) (10, -0.305885 + 0.580036 i) (11,
1.23543 - 2.20782 i)

Both GAMG and ILU are nice and dandy for this, but as soon as I look at a
bigger system, like a network with 8500 buses, the out-of-box gamg craps
out. I am not sure where to start when it comes to tuning GAMG. Attached is
the ksp monitor/view output for gamg on the unsuccessful solve

I'm also attaching a zip file which contains the simple PETSc script that
loads the binary matrix/vector as well as two test cases, if you guys are
interested in trying it out. It only works if you have PETSc configured
with complex numbers.

Thanks

Justin

PS - A couple years ago I had asked if there was a paper/tutorial on
using/tuning GAMG. Does such a thing exist today?

On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:
>
>> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>>
>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>
>> Right now we're just trying to solve eq(5) (in section 1), inverting the
>> linear Y-bus matrix. Eventually we have to be able to solve equations like
>> those in the next section.
>>
>
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix
> to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You
> can start out just doing relaxation, like SOR, on
> small examples.
>
>   Thanks,
>
>     Matt
>
>
>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm working with some folks to extract a linear system of equations
>>>> from an external software package that solves power flow equations in
>>>> complex form. Since that external package uses serial direct solvers like
>>>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>>>> be solved in PETSc using its parallel solvers.
>>>>
>>>> I got mumps to achieve a very minor speedup across two MPI processes on
>>>> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>>>> seconds). However I want to use iterative solvers and preconditioners but I
>>>> have never worked with complex numbers so I am not sure what the "best"
>>>> options are given PETSc's capabilities.
>>>>
>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>> converged slowly. Does anyone have recommendations on how one would go
>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>> claim to successfully precondition power flow equations operate in this
>>>> form.
>>>>
>>>
>>> 1) We really need to see the (simplified) equations
>>>
>>> 2) All complex equations can be converted to a system of real equations
>>> twice as large, but this is not necessarily the best way to go
>>>
>>>  Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Justin
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/05296caa/attachment-0001.html>
-------------- next part --------------
  0 KSP preconditioned resid norm 1.556733784759e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 2.467734198385e+02 true resid norm 6.926512200810e+04 ||r(i)||/||b|| 5.736241988246e-04
  2 KSP preconditioned resid norm 1.035114767491e+01 true resid norm 1.173822977303e+04 ||r(i)||/||b|| 9.721101261303e-05
  3 KSP preconditioned resid norm 9.783840856528e+00 true resid norm 1.174200950657e+04 ||r(i)||/||b|| 9.724231475421e-05
  4 KSP preconditioned resid norm 9.697460582239e+00 true resid norm 1.170657456347e+04 ||r(i)||/||b|| 9.694885766843e-05
  5 KSP preconditioned resid norm 9.624514877355e+00 true resid norm 1.171823270946e+04 ||r(i)||/||b|| 9.704540546130e-05
  6 KSP preconditioned resid norm 9.561349759033e+00 true resid norm 1.172489502718e+04 ||r(i)||/||b|| 9.710057993517e-05
  7 KSP preconditioned resid norm 9.534271479761e+00 true resid norm 1.173251828014e+04 ||r(i)||/||b|| 9.716371246490e-05
  8 KSP preconditioned resid norm 9.450575125350e+00 true resid norm 1.172667482941e+04 ||r(i)||/||b|| 9.711531949815e-05
  9 KSP preconditioned resid norm 9.348923573226e+00 true resid norm 1.172215128289e+04 ||r(i)||/||b|| 9.707785741520e-05
 10 KSP preconditioned resid norm 9.275514422035e+00 true resid norm 1.172820845896e+04 ||r(i)||/||b|| 9.712802036403e-05
 11 KSP preconditioned resid norm 9.247308197103e+00 true resid norm 1.170632318348e+04 ||r(i)||/||b|| 9.694677584660e-05
 12 KSP preconditioned resid norm 9.242402347896e+00 true resid norm 1.170746812872e+04 ||r(i)||/||b|| 9.695625779472e-05
 13 KSP preconditioned resid norm 9.124679268608e+00 true resid norm 1.168734027045e+04 ||r(i)||/||b|| 9.678956745711e-05
 14 KSP preconditioned resid norm 9.055354678228e+00 true resid norm 1.166944673061e+04 ||r(i)||/||b|| 9.664138079173e-05
 15 KSP preconditioned resid norm 9.037215864513e+00 true resid norm 1.165198272950e+04 ||r(i)||/||b|| 9.649675138296e-05
 16 KSP preconditioned resid norm 9.010603910954e+00 true resid norm 1.166468564222e+04 ||r(i)||/||b|| 9.660195148831e-05
 17 KSP preconditioned resid norm 8.981244422498e+00 true resid norm 1.168667191283e+04 ||r(i)||/||b|| 9.678403240437e-05
 18 KSP preconditioned resid norm 8.979510503100e+00 true resid norm 1.168307146612e+04 ||r(i)||/||b|| 9.675421504026e-05
 19 KSP preconditioned resid norm 8.968878344461e+00 true resid norm 1.169221155863e+04 ||r(i)||/||b|| 9.682990938818e-05
 20 KSP preconditioned resid norm 8.967010985664e+00 true resid norm 1.169713694003e+04 ||r(i)||/||b|| 9.687069929624e-05
 21 KSP preconditioned resid norm 8.965914406739e+00 true resid norm 1.169447418295e+04 ||r(i)||/||b|| 9.684864747784e-05
 22 KSP preconditioned resid norm 8.961184626034e+00 true resid norm 1.168812627068e+04 ||r(i)||/||b|| 9.679607677575e-05
 23 KSP preconditioned resid norm 8.950303182756e+00 true resid norm 1.166443979772e+04 ||r(i)||/||b|| 9.659991550906e-05
 24 KSP preconditioned resid norm 8.920807263658e+00 true resid norm 1.162218866637e+04 ||r(i)||/||b|| 9.625000965933e-05
 25 KSP preconditioned resid norm 8.867603836991e+00 true resid norm 1.157905917259e+04 ||r(i)||/||b|| 9.589282958663e-05
 26 KSP preconditioned resid norm 8.826316257889e+00 true resid norm 1.155070910214e+04 ||r(i)||/||b|| 9.565804639449e-05
 27 KSP preconditioned resid norm 8.717737070764e+00 true resid norm 1.148758744724e+04 ||r(i)||/||b|| 9.513529977008e-05
 28 KSP preconditioned resid norm 8.715478718117e+00 true resid norm 1.149653735948e+04 ||r(i)||/||b|| 9.520941912611e-05
 29 KSP preconditioned resid norm 8.663250781861e+00 true resid norm 1.144212407512e+04 ||r(i)||/||b|| 9.475879151231e-05
 30 KSP preconditioned resid norm 8.650224227426e+00 true resid norm 1.141693447064e+04 ||r(i)||/||b|| 9.455018195140e-05
 31 KSP preconditioned resid norm 8.650179905089e+00 true resid norm 1.141719804031e+04 ||r(i)||/||b|| 9.455236472302e-05
 32 KSP preconditioned resid norm 8.648700220300e+00 true resid norm 1.141843356273e+04 ||r(i)||/||b|| 9.456259679277e-05
 33 KSP preconditioned resid norm 8.647618018772e+00 true resid norm 1.141813089857e+04 ||r(i)||/||b|| 9.456009025726e-05
 34 KSP preconditioned resid norm 8.647220857775e+00 true resid norm 1.141896841019e+04 ||r(i)||/||b|| 9.456702617133e-05
 35 KSP preconditioned resid norm 8.643958615519e+00 true resid norm 1.142025753010e+04 ||r(i)||/||b|| 9.457770211257e-05
 36 KSP preconditioned resid norm 8.635563807844e+00 true resid norm 1.141889814360e+04 ||r(i)||/||b|| 9.456644425335e-05
 37 KSP preconditioned resid norm 8.630435670335e+00 true resid norm 1.141914255411e+04 ||r(i)||/||b|| 9.456846835698e-05
 38 KSP preconditioned resid norm 8.624297760958e+00 true resid norm 1.142303833051e+04 ||r(i)||/||b|| 9.460073151558e-05
 39 KSP preconditioned resid norm 8.615427625780e+00 true resid norm 1.142117191476e+04 ||r(i)||/||b|| 9.458527465637e-05
 40 KSP preconditioned resid norm 8.609524393404e+00 true resid norm 1.142356577778e+04 ||r(i)||/||b|| 9.460509960887e-05
 41 KSP preconditioned resid norm 8.605062228984e+00 true resid norm 1.142481336207e+04 ||r(i)||/||b|| 9.461543156988e-05
 42 KSP preconditioned resid norm 8.600640170257e+00 true resid norm 1.142717590694e+04 ||r(i)||/||b|| 9.463499715886e-05
 43 KSP preconditioned resid norm 8.592288313234e+00 true resid norm 1.142228040106e+04 ||r(i)||/||b|| 9.459445466711e-05
 44 KSP preconditioned resid norm 8.589775051997e+00 true resid norm 1.142665501967e+04 ||r(i)||/||b|| 9.463068339265e-05
 45 KSP preconditioned resid norm 8.588224195960e+00 true resid norm 1.142311984607e+04 ||r(i)||/||b|| 9.460140659269e-05
 46 KSP preconditioned resid norm 8.586641173460e+00 true resid norm 1.142761000938e+04 ||r(i)||/||b|| 9.463859221013e-05
 47 KSP preconditioned resid norm 8.582843003591e+00 true resid norm 1.142820973251e+04 ||r(i)||/||b|| 9.464355886133e-05
 48 KSP preconditioned resid norm 8.578114120299e+00 true resid norm 1.142026973821e+04 ||r(i)||/||b|| 9.457780321498e-05
 49 KSP preconditioned resid norm 8.552066003429e+00 true resid norm 1.143356645998e+04 ||r(i)||/||b|| 9.468792099355e-05
 50 KSP preconditioned resid norm 8.522429597744e+00 true resid norm 1.143428431491e+04 ||r(i)||/||b|| 9.469386596194e-05
 51 KSP preconditioned resid norm 8.509971142515e+00 true resid norm 1.143169998447e+04 ||r(i)||/||b|| 9.467246363950e-05
 52 KSP preconditioned resid norm 8.502914104699e+00 true resid norm 1.144980387842e+04 ||r(i)||/||b|| 9.482239236785e-05
 53 KSP preconditioned resid norm 8.497768053698e+00 true resid norm 1.143338107677e+04 ||r(i)||/||b|| 9.468638572889e-05
 54 KSP preconditioned resid norm 8.486612717104e+00 true resid norm 1.145266065638e+04 ||r(i)||/||b|| 9.484605098449e-05
 55 KSP preconditioned resid norm 8.421723991813e+00 true resid norm 1.143942489297e+04 ||r(i)||/||b|| 9.473643803700e-05
 56 KSP preconditioned resid norm 8.411554038997e+00 true resid norm 1.146640523851e+04 ||r(i)||/||b|| 9.495987775158e-05
 57 KSP preconditioned resid norm 8.396885677701e+00 true resid norm 1.145255286778e+04 ||r(i)||/||b|| 9.484515832524e-05
 58 KSP preconditioned resid norm 8.392398200322e+00 true resid norm 1.144436262136e+04 ||r(i)||/||b|| 9.477733019753e-05
 59 KSP preconditioned resid norm 8.387319386374e+00 true resid norm 1.145926595392e+04 ||r(i)||/||b|| 9.490075324153e-05
 60 KSP preconditioned resid norm 8.275168546170e+00 true resid norm 1.143802695582e+04 ||r(i)||/||b|| 9.472486091771e-05
 61 KSP preconditioned resid norm 8.275167866382e+00 true resid norm 1.143795381204e+04 ||r(i)||/||b|| 9.472425517215e-05
 62 KSP preconditioned resid norm 8.272316803128e+00 true resid norm 1.143541865677e+04 ||r(i)||/||b|| 9.470326009743e-05
 63 KSP preconditioned resid norm 8.261583354759e+00 true resid norm 1.144068619665e+04 ||r(i)||/||b|| 9.474688361614e-05
 64 KSP preconditioned resid norm 8.241071325522e+00 true resid norm 1.144258105031e+04 ||r(i)||/||b|| 9.476257598601e-05
 65 KSP preconditioned resid norm 8.236391632636e+00 true resid norm 1.144430171734e+04 ||r(i)||/||b|| 9.477682581644e-05
 66 KSP preconditioned resid norm 8.229070077318e+00 true resid norm 1.144172893861e+04 ||r(i)||/||b|| 9.475551916031e-05
 67 KSP preconditioned resid norm 8.227797974684e+00 true resid norm 1.144266693609e+04 ||r(i)||/||b|| 9.476328725541e-05
 68 KSP preconditioned resid norm 8.215516740033e+00 true resid norm 1.145001198123e+04 ||r(i)||/||b|| 9.482411578659e-05
 69 KSP preconditioned resid norm 8.193687542733e+00 true resid norm 1.144482368537e+04 ||r(i)||/||b|| 9.478114853303e-05
 70 KSP preconditioned resid norm 8.185832966219e+00 true resid norm 1.145141909518e+04 ||r(i)||/||b|| 9.483576890417e-05
 71 KSP preconditioned resid norm 8.179747707886e+00 true resid norm 1.144869851280e+04 ||r(i)||/||b|| 9.481323820121e-05
 72 KSP preconditioned resid norm 8.170346432637e+00 true resid norm 1.145587113928e+04 ||r(i)||/||b|| 9.487263883458e-05
 73 KSP preconditioned resid norm 8.159202384369e+00 true resid norm 1.144924368304e+04 ||r(i)||/||b|| 9.481775306864e-05
 74 KSP preconditioned resid norm 8.146859593910e+00 true resid norm 1.146047121901e+04 ||r(i)||/||b|| 9.491073473293e-05
 75 KSP preconditioned resid norm 8.137832991741e+00 true resid norm 1.145300608694e+04 ||r(i)||/||b|| 9.484891169301e-05
 76 KSP preconditioned resid norm 8.135515335248e+00 true resid norm 1.145585787424e+04 ||r(i)||/||b|| 9.487252897922e-05
 77 KSP preconditioned resid norm 8.132043073553e+00 true resid norm 1.145804395405e+04 ||r(i)||/||b|| 9.489063315984e-05
 78 KSP preconditioned resid norm 8.131639278793e+00 true resid norm 1.145542925757e+04 ||r(i)||/||b|| 9.486897935869e-05
 79 KSP preconditioned resid norm 8.129443623048e+00 true resid norm 1.145386155906e+04 ||r(i)||/||b|| 9.485599634832e-05
 80 KSP preconditioned resid norm 8.122103138067e+00 true resid norm 1.145506982500e+04 ||r(i)||/||b|| 9.486600269149e-05
 81 KSP preconditioned resid norm 8.111955169975e+00 true resid norm 1.146666195464e+04 ||r(i)||/||b|| 9.496200376512e-05
 82 KSP preconditioned resid norm 8.107349030577e+00 true resid norm 1.146486925676e+04 ||r(i)||/||b|| 9.494715740586e-05
 83 KSP preconditioned resid norm 8.105070007503e+00 true resid norm 1.147505327554e+04 ||r(i)||/||b|| 9.503149710587e-05
 84 KSP preconditioned resid norm 8.103544131106e+00 true resid norm 1.146874745520e+04 ||r(i)||/||b|| 9.497927499125e-05
 85 KSP preconditioned resid norm 8.099735884338e+00 true resid norm 1.146182117236e+04 ||r(i)||/||b|| 9.492191447087e-05
 86 KSP preconditioned resid norm 8.098066017143e+00 true resid norm 1.145125305602e+04 ||r(i)||/||b|| 9.483439383861e-05
 87 KSP preconditioned resid norm 8.089179556223e+00 true resid norm 1.146416654622e+04 ||r(i)||/||b|| 9.494133785690e-05
 88 KSP preconditioned resid norm 8.087711130217e+00 true resid norm 1.146753999073e+04 ||r(i)||/||b|| 9.496927528551e-05
 89 KSP preconditioned resid norm 8.072162312704e+00 true resid norm 1.147368481011e+04 ||r(i)||/||b|| 9.502016405887e-05
 90 KSP preconditioned resid norm 8.069914145869e+00 true resid norm 1.146514526975e+04 ||r(i)||/||b|| 9.494944322772e-05
 91 KSP preconditioned resid norm 8.069912058750e+00 true resid norm 1.146493412240e+04 ||r(i)||/||b|| 9.494769459544e-05
 92 KSP preconditioned resid norm 8.069751497433e+00 true resid norm 1.146490715288e+04 ||r(i)||/||b|| 9.494747124539e-05
 93 KSP preconditioned resid norm 8.068911887066e+00 true resid norm 1.146541882213e+04 ||r(i)||/||b|| 9.495170867182e-05
 94 KSP preconditioned resid norm 8.068294163812e+00 true resid norm 1.146547318034e+04 ||r(i)||/||b|| 9.495215884335e-05
 95 KSP preconditioned resid norm 8.067330245843e+00 true resid norm 1.146681341377e+04 ||r(i)||/||b|| 9.496325808505e-05
 96 KSP preconditioned resid norm 8.066642564144e+00 true resid norm 1.146520657725e+04 ||r(i)||/||b|| 9.494995095025e-05
 97 KSP preconditioned resid norm 8.066580436781e+00 true resid norm 1.146496901743e+04 ||r(i)||/||b|| 9.494798358113e-05
 98 KSP preconditioned resid norm 8.062536068911e+00 true resid norm 1.146824401427e+04 ||r(i)||/||b|| 9.497510570824e-05
 99 KSP preconditioned resid norm 8.059112974378e+00 true resid norm 1.146563252712e+04 ||r(i)||/||b|| 9.495347848540e-05
100 KSP preconditioned resid norm 8.058522171195e+00 true resid norm 1.146702283797e+04 ||r(i)||/||b|| 9.496499244695e-05
101 KSP preconditioned resid norm 8.058411636082e+00 true resid norm 1.146646239883e+04 ||r(i)||/||b|| 9.496035112899e-05
102 KSP preconditioned resid norm 8.057901996972e+00 true resid norm 1.146787924868e+04 ||r(i)||/||b|| 9.497208487519e-05
103 KSP preconditioned resid norm 8.055964747615e+00 true resid norm 1.146469247142e+04 ||r(i)||/||b|| 9.494569334502e-05
104 KSP preconditioned resid norm 8.053325795070e+00 true resid norm 1.146923378183e+04 ||r(i)||/||b|| 9.498330254099e-05
105 KSP preconditioned resid norm 8.053272828449e+00 true resid norm 1.146934053046e+04 ||r(i)||/||b|| 9.498418658759e-05
106 KSP preconditioned resid norm 8.050001476934e+00 true resid norm 1.146379981140e+04 ||r(i)||/||b|| 9.493830071550e-05
107 KSP preconditioned resid norm 8.042356505342e+00 true resid norm 1.147674978091e+04 ||r(i)||/||b|| 9.504554683983e-05
108 KSP preconditioned resid norm 8.038304252203e+00 true resid norm 1.147061999962e+04 ||r(i)||/||b|| 9.499478260554e-05
109 KSP preconditioned resid norm 8.034571690825e+00 true resid norm 1.147861914609e+04 ||r(i)||/||b|| 9.506102812490e-05
110 KSP preconditioned resid norm 8.027279100787e+00 true resid norm 1.146865983786e+04 ||r(i)||/||b|| 9.497854938181e-05
111 KSP preconditioned resid norm 8.019309959719e+00 true resid norm 1.147771668707e+04 ||r(i)||/||b|| 9.505355434421e-05
112 KSP preconditioned resid norm 8.018444183552e+00 true resid norm 1.147277743510e+04 ||r(i)||/||b|| 9.501264956602e-05
113 KSP preconditioned resid norm 8.018314507549e+00 true resid norm 1.147321254112e+04 ||r(i)||/||b|| 9.501625292846e-05
114 KSP preconditioned resid norm 8.018166328378e+00 true resid norm 1.147043656341e+04 ||r(i)||/||b|| 9.499326346508e-05
115 KSP preconditioned resid norm 8.017791284963e+00 true resid norm 1.146694594216e+04 ||r(i)||/||b|| 9.496435562866e-05
116 KSP preconditioned resid norm 8.016831568897e+00 true resid norm 1.146926041550e+04 ||r(i)||/||b|| 9.498352310966e-05
117 KSP preconditioned resid norm 8.003250614397e+00 true resid norm 1.147324612478e+04 ||r(i)||/||b|| 9.501653105408e-05
118 KSP preconditioned resid norm 7.978267535264e+00 true resid norm 1.147602339297e+04 ||r(i)||/||b|| 9.503953120467e-05
119 KSP preconditioned resid norm 7.963775301980e+00 true resid norm 1.148997544004e+04 ||r(i)||/||b|| 9.515507610795e-05
120 KSP preconditioned resid norm 7.957040961860e+00 true resid norm 1.147327720922e+04 ||r(i)||/||b|| 9.501678848214e-05
121 KSP preconditioned resid norm 7.957039915526e+00 true resid norm 1.147343436833e+04 ||r(i)||/||b|| 9.501809000686e-05
122 KSP preconditioned resid norm 7.956854567631e+00 true resid norm 1.147280966261e+04 ||r(i)||/||b|| 9.501291646051e-05
123 KSP preconditioned resid norm 7.955245135609e+00 true resid norm 1.147551816032e+04 ||r(i)||/||b|| 9.503534708335e-05
124 KSP preconditioned resid norm 7.949424288407e+00 true resid norm 1.147338665139e+04 ||r(i)||/||b|| 9.501769483548e-05
125 KSP preconditioned resid norm 7.935530789816e+00 true resid norm 1.147874056934e+04 ||r(i)||/||b|| 9.506203370053e-05
126 KSP preconditioned resid norm 7.927536005194e+00 true resid norm 1.147392190112e+04 ||r(i)||/||b|| 9.502212754549e-05
127 KSP preconditioned resid norm 7.926101716355e+00 true resid norm 1.147718844520e+04 ||r(i)||/||b|| 9.504917967033e-05
128 KSP preconditioned resid norm 7.925228572070e+00 true resid norm 1.147783852581e+04 ||r(i)||/||b|| 9.505456336074e-05
129 KSP preconditioned resid norm 7.924576209081e+00 true resid norm 1.147669458848e+04 ||r(i)||/||b|| 9.504508975964e-05
130 KSP preconditioned resid norm 7.922615866279e+00 true resid norm 1.147925283706e+04 ||r(i)||/||b|| 9.506627608331e-05
131 KSP preconditioned resid norm 7.918779515591e+00 true resid norm 1.147713281143e+04 ||r(i)||/||b|| 9.504871893521e-05
132 KSP preconditioned resid norm 7.909412907788e+00 true resid norm 1.148297762598e+04 ||r(i)||/||b|| 9.509712319645e-05
133 KSP preconditioned resid norm 7.899763396373e+00 true resid norm 1.147789903283e+04 ||r(i)||/||b|| 9.505506445401e-05
134 KSP preconditioned resid norm 7.892690216660e+00 true resid norm 1.148660270211e+04 ||r(i)||/||b|| 9.512714453092e-05
135 KSP preconditioned resid norm 7.888927423775e+00 true resid norm 1.148323921609e+04 ||r(i)||/||b|| 9.509928957422e-05
136 KSP preconditioned resid norm 7.885769001852e+00 true resid norm 1.148731605561e+04 ||r(i)||/||b|| 9.513305222033e-05
137 KSP preconditioned resid norm 7.881117593851e+00 true resid norm 1.147894351690e+04 ||r(i)||/||b|| 9.506371442566e-05
138 KSP preconditioned resid norm 7.878372902094e+00 true resid norm 1.148291345879e+04 ||r(i)||/||b|| 9.509659179122e-05
139 KSP preconditioned resid norm 7.877693657380e+00 true resid norm 1.148389413972e+04 ||r(i)||/||b|| 9.510471337238e-05
140 KSP preconditioned resid norm 7.874339819009e+00 true resid norm 1.147897833272e+04 ||r(i)||/||b|| 9.506400275537e-05
141 KSP preconditioned resid norm 7.870169549296e+00 true resid norm 1.148885777837e+04 ||r(i)||/||b|| 9.514582011068e-05
142 KSP preconditioned resid norm 7.865065520311e+00 true resid norm 1.149330087122e+04 ||r(i)||/||b|| 9.518261591072e-05
143 KSP preconditioned resid norm 7.861109764040e+00 true resid norm 1.148012084307e+04 ||r(i)||/||b|| 9.507346453882e-05
144 KSP preconditioned resid norm 7.860435412646e+00 true resid norm 1.148435390954e+04 ||r(i)||/||b|| 9.510852098997e-05
145 KSP preconditioned resid norm 7.855828192977e+00 true resid norm 1.149276842179e+04 ||r(i)||/||b|| 9.517820639161e-05
146 KSP preconditioned resid norm 7.850170188526e+00 true resid norm 1.148948238725e+04 ||r(i)||/||b|| 9.515099285505e-05
147 KSP preconditioned resid norm 7.849676187999e+00 true resid norm 1.149484747891e+04 ||r(i)||/||b|| 9.519542425595e-05
148 KSP preconditioned resid norm 7.846734599671e+00 true resid norm 1.150142940119e+04 ||r(i)||/||b|| 9.524993292905e-05
149 KSP preconditioned resid norm 7.840328210575e+00 true resid norm 1.149865564794e+04 ||r(i)||/||b|| 9.522696188765e-05
150 KSP preconditioned resid norm 7.837868671850e+00 true resid norm 1.148731412803e+04 ||r(i)||/||b|| 9.513303625696e-05
151 KSP preconditioned resid norm 7.837868612857e+00 true resid norm 1.148733090409e+04 ||r(i)||/||b|| 9.513317518912e-05
152 KSP preconditioned resid norm 7.837867474044e+00 true resid norm 1.148728129363e+04 ||r(i)||/||b|| 9.513276433642e-05
153 KSP preconditioned resid norm 7.837752725804e+00 true resid norm 1.148795140128e+04 ||r(i)||/||b|| 9.513831388222e-05
154 KSP preconditioned resid norm 7.836765314081e+00 true resid norm 1.148702415252e+04 ||r(i)||/||b|| 9.513063480347e-05
155 KSP preconditioned resid norm 7.833365964413e+00 true resid norm 1.148939147946e+04 ||r(i)||/||b|| 9.515023999552e-05
156 KSP preconditioned resid norm 7.832032890301e+00 true resid norm 1.148713750037e+04 ||r(i)||/||b|| 9.513157350202e-05
157 KSP preconditioned resid norm 7.831864174530e+00 true resid norm 1.148629901415e+04 ||r(i)||/||b|| 9.512462951673e-05
158 KSP preconditioned resid norm 7.829104913744e+00 true resid norm 1.148876006232e+04 ||r(i)||/||b|| 9.514501086802e-05
159 KSP preconditioned resid norm 7.827574187011e+00 true resid norm 1.148761084781e+04 ||r(i)||/||b|| 9.513549356362e-05
160 KSP preconditioned resid norm 7.827367990924e+00 true resid norm 1.148829273935e+04 ||r(i)||/||b|| 9.514114069853e-05
161 KSP preconditioned resid norm 7.826777316123e+00 true resid norm 1.148745697083e+04 ||r(i)||/||b|| 9.513421922005e-05
162 KSP preconditioned resid norm 7.824670116085e+00 true resid norm 1.149111907676e+04 ||r(i)||/||b|| 9.516454721953e-05
163 KSP preconditioned resid norm 7.820538776708e+00 true resid norm 1.148674443883e+04 ||r(i)||/||b|| 9.512831833396e-05
164 KSP preconditioned resid norm 7.816254945507e+00 true resid norm 1.149360623129e+04 ||r(i)||/||b|| 9.518514477252e-05
165 KSP preconditioned resid norm 7.814786322960e+00 true resid norm 1.149050417748e+04 ||r(i)||/||b|| 9.515945488592e-05
166 KSP preconditioned resid norm 7.813777026542e+00 true resid norm 1.148713734136e+04 ||r(i)||/||b|| 9.513157218515e-05
167 KSP preconditioned resid norm 7.808480079332e+00 true resid norm 1.149732820642e+04 ||r(i)||/||b|| 9.521596858313e-05
168 KSP preconditioned resid norm 7.803270036211e+00 true resid norm 1.148967484720e+04 ||r(i)||/||b|| 9.515258672629e-05
169 KSP preconditioned resid norm 7.800393672452e+00 true resid norm 1.149689289845e+04 ||r(i)||/||b|| 9.521236354820e-05
170 KSP preconditioned resid norm 7.794521000187e+00 true resid norm 1.148750868788e+04 ||r(i)||/||b|| 9.513464751864e-05
171 KSP preconditioned resid norm 7.783160138176e+00 true resid norm 1.150656300296e+04 ||r(i)||/||b|| 9.529244722945e-05
172 KSP preconditioned resid norm 7.776696889938e+00 true resid norm 1.149853021988e+04 ||r(i)||/||b|| 9.522592314594e-05
173 KSP preconditioned resid norm 7.774128340987e+00 true resid norm 1.149763894508e+04 ||r(i)||/||b|| 9.521854198816e-05
174 KSP preconditioned resid norm 7.770382044787e+00 true resid norm 1.149192924947e+04 ||r(i)||/||b|| 9.517125672435e-05
175 KSP preconditioned resid norm 7.763268399296e+00 true resid norm 1.149798854904e+04 ||r(i)||/||b|| 9.522143725907e-05
176 KSP preconditioned resid norm 7.760867895366e+00 true resid norm 1.149910039906e+04 ||r(i)||/||b|| 9.523064512671e-05
177 KSP preconditioned resid norm 7.758180070841e+00 true resid norm 1.149705763701e+04 ||r(i)||/||b|| 9.521372784268e-05
178 KSP preconditioned resid norm 7.742236443400e+00 true resid norm 1.149305947334e+04 ||r(i)||/||b|| 9.518061675636e-05
179 KSP preconditioned resid norm 7.724097411312e+00 true resid norm 1.150935242631e+04 ||r(i)||/||b|| 9.531554804392e-05
180 KSP preconditioned resid norm 7.710284305033e+00 true resid norm 1.150181366069e+04 ||r(i)||/||b|| 9.525311520236e-05
181 KSP preconditioned resid norm 7.710281251601e+00 true resid norm 1.150202074537e+04 ||r(i)||/||b|| 9.525483018939e-05
182 KSP preconditioned resid norm 7.709864202876e+00 true resid norm 1.150143814781e+04 ||r(i)||/||b|| 9.525000536487e-05
183 KSP preconditioned resid norm 7.706452833465e+00 true resid norm 1.150555049867e+04 ||r(i)||/||b|| 9.528406210078e-05
184 KSP preconditioned resid norm 7.698298665533e+00 true resid norm 1.150308993468e+04 ||r(i)||/||b|| 9.526368475920e-05
185 KSP preconditioned resid norm 7.686132042497e+00 true resid norm 1.150750153898e+04 ||r(i)||/||b|| 9.530021978446e-05
186 KSP preconditioned resid norm 7.678925550590e+00 true resid norm 1.150331945881e+04 ||r(i)||/||b|| 9.526558558016e-05
187 KSP preconditioned resid norm 7.678492265313e+00 true resid norm 1.150497573345e+04 ||r(i)||/||b|| 9.527930214031e-05
188 KSP preconditioned resid norm 7.676827247627e+00 true resid norm 1.150647440506e+04 ||r(i)||/||b|| 9.529171349946e-05
189 KSP preconditioned resid norm 7.675502915492e+00 true resid norm 1.150496654661e+04 ||r(i)||/||b|| 9.527922605882e-05
190 KSP preconditioned resid norm 7.671144082061e+00 true resid norm 1.150973939819e+04 ||r(i)||/||b|| 9.531875278004e-05
191 KSP preconditioned resid norm 7.664867711248e+00 true resid norm 1.150678903619e+04 ||r(i)||/||b|| 9.529431914027e-05
192 KSP preconditioned resid norm 7.656980277053e+00 true resid norm 1.151278367351e+04 ||r(i)||/||b|| 9.534396416986e-05
193 KSP preconditioned resid norm 7.651285979353e+00 true resid norm 1.150772431407e+04 ||r(i)||/||b|| 9.530206471275e-05
194 KSP preconditioned resid norm 7.644892771578e+00 true resid norm 1.151536668474e+04 ||r(i)||/||b|| 9.536535556718e-05
195 KSP preconditioned resid norm 7.639723688192e+00 true resid norm 1.151080389241e+04 ||r(i)||/||b|| 9.532756846713e-05
196 KSP preconditioned resid norm 7.636682020526e+00 true resid norm 1.151657682541e+04 ||r(i)||/||b|| 9.537537743607e-05
197 KSP preconditioned resid norm 7.634678946384e+00 true resid norm 1.151227379077e+04 ||r(i)||/||b|| 9.533974153846e-05
198 KSP preconditioned resid norm 7.632604899482e+00 true resid norm 1.151396525698e+04 ||r(i)||/||b|| 9.535374954019e-05
199 KSP preconditioned resid norm 7.632492673218e+00 true resid norm 1.151523319461e+04 ||r(i)||/||b|| 9.536425005888e-05
200 KSP preconditioned resid norm 7.631110380087e+00 true resid norm 1.151021049096e+04 ||r(i)||/||b|| 9.532265416939e-05
201 KSP preconditioned resid norm 7.625261586004e+00 true resid norm 1.152103590995e+04 ||r(i)||/||b|| 9.541230567244e-05
202 KSP preconditioned resid norm 7.619726670713e+00 true resid norm 1.152045313199e+04 ||r(i)||/||b|| 9.540747935392e-05
203 KSP preconditioned resid norm 7.617878598792e+00 true resid norm 1.151347601705e+04 ||r(i)||/||b|| 9.534969786375e-05
204 KSP preconditioned resid norm 7.616575311171e+00 true resid norm 1.151515309398e+04 ||r(i)||/||b|| 9.536358669962e-05
205 KSP preconditioned resid norm 7.614919925880e+00 true resid norm 1.152161323195e+04 ||r(i)||/||b|| 9.541708680696e-05
206 KSP preconditioned resid norm 7.611717803156e+00 true resid norm 1.152651413057e+04 ||r(i)||/||b|| 9.545767395914e-05
207 KSP preconditioned resid norm 7.608328146076e+00 true resid norm 1.152242569336e+04 ||r(i)||/||b|| 9.542381526591e-05
208 KSP preconditioned resid norm 7.607585027893e+00 true resid norm 1.152908476395e+04 ||r(i)||/||b|| 9.547896284837e-05
209 KSP preconditioned resid norm 7.605189547370e+00 true resid norm 1.152857356953e+04 ||r(i)||/||b|| 9.547472935421e-05
210 KSP preconditioned resid norm 7.601043570412e+00 true resid norm 1.151611810055e+04 ||r(i)||/||b|| 9.537157847244e-05
211 KSP preconditioned resid norm 7.601043409439e+00 true resid norm 1.151612682623e+04 ||r(i)||/||b|| 9.537165073482e-05
212 KSP preconditioned resid norm 7.601033094834e+00 true resid norm 1.151624219804e+04 ||r(i)||/||b|| 9.537260619488e-05
213 KSP preconditioned resid norm 7.600909216842e+00 true resid norm 1.151651088859e+04 ||r(i)||/||b|| 9.537483137545e-05
214 KSP preconditioned resid norm 7.600332596796e+00 true resid norm 1.151582000207e+04 ||r(i)||/||b|| 9.536910974799e-05
215 KSP preconditioned resid norm 7.598970465279e+00 true resid norm 1.151711525876e+04 ||r(i)||/||b|| 9.537983651145e-05
216 KSP preconditioned resid norm 7.597916907452e+00 true resid norm 1.151536467897e+04 ||r(i)||/||b|| 9.536533895625e-05
217 KSP preconditioned resid norm 7.597535123360e+00 true resid norm 1.151470755036e+04 ||r(i)||/||b|| 9.535989689736e-05
218 KSP preconditioned resid norm 7.595946799867e+00 true resid norm 1.151654956676e+04 ||r(i)||/||b|| 9.537515169155e-05
219 KSP preconditioned resid norm 7.594754973227e+00 true resid norm 1.151498699305e+04 ||r(i)||/||b|| 9.536221112252e-05
220 KSP preconditioned resid norm 7.593800163770e+00 true resid norm 1.151694663616e+04 ||r(i)||/||b|| 9.537844005099e-05
221 KSP preconditioned resid norm 7.593208994376e+00 true resid norm 1.151610072510e+04 ||r(i)||/||b|| 9.537143457639e-05
222 KSP preconditioned resid norm 7.591914504687e+00 true resid norm 1.151826922909e+04 ||r(i)||/||b|| 9.538939320151e-05
223 KSP preconditioned resid norm 7.590332302884e+00 true resid norm 1.151610590371e+04 ||r(i)||/||b|| 9.537147746340e-05
224 KSP preconditioned resid norm 7.586053769386e+00 true resid norm 1.152227409030e+04 ||r(i)||/||b|| 9.542255975401e-05
225 KSP preconditioned resid norm 7.580716222153e+00 true resid norm 1.151683117324e+04 ||r(i)||/||b|| 9.537748383636e-05
226 KSP preconditioned resid norm 7.580289714570e+00 true resid norm 1.151877890336e+04 ||r(i)||/||b|| 9.539361410649e-05
227 KSP preconditioned resid norm 7.578947436655e+00 true resid norm 1.152195528985e+04 ||r(i)||/||b|| 9.541991958463e-05
228 KSP preconditioned resid norm 7.576299048296e+00 true resid norm 1.151732150446e+04 ||r(i)||/||b|| 9.538154455032e-05
229 KSP preconditioned resid norm 7.572861273503e+00 true resid norm 1.152626205331e+04 ||r(i)||/||b|| 9.545558636278e-05
230 KSP preconditioned resid norm 7.569880814692e+00 true resid norm 1.151771625206e+04 ||r(i)||/||b|| 9.538481368163e-05
231 KSP preconditioned resid norm 7.565899990960e+00 true resid norm 1.152985557489e+04 ||r(i)||/||b|| 9.548534637588e-05
232 KSP preconditioned resid norm 7.560505548643e+00 true resid norm 1.151796624266e+04 ||r(i)||/||b|| 9.538688399719e-05
233 KSP preconditioned resid norm 7.555467398510e+00 true resid norm 1.152607191292e+04 ||r(i)||/||b|| 9.545401170118e-05
234 KSP preconditioned resid norm 7.550249680153e+00 true resid norm 1.151974095673e+04 ||r(i)||/||b|| 9.540158142220e-05
235 KSP preconditioned resid norm 7.548215439327e+00 true resid norm 1.152541627738e+04 ||r(i)||/||b|| 9.544858200729e-05
236 KSP preconditioned resid norm 7.545003747197e+00 true resid norm 1.152785431496e+04 ||r(i)||/||b|| 9.546877279464e-05
237 KSP preconditioned resid norm 7.541591990440e+00 true resid norm 1.152164094498e+04 ||r(i)||/||b|| 9.541731631449e-05
238 KSP preconditioned resid norm 7.540246454073e+00 true resid norm 1.152531974471e+04 ||r(i)||/||b|| 9.544778256482e-05
239 KSP preconditioned resid norm 7.534266026574e+00 true resid norm 1.153518804438e+04 ||r(i)||/||b|| 9.552950761389e-05
240 KSP preconditioned resid norm 7.524136196748e+00 true resid norm 1.151327096014e+04 ||r(i)||/||b|| 9.534799966986e-05
241 KSP preconditioned resid norm 7.524134491328e+00 true resid norm 1.151345946406e+04 ||r(i)||/||b|| 9.534956077895e-05
242 KSP preconditioned resid norm 7.524112716097e+00 true resid norm 1.151331483040e+04 ||r(i)||/||b|| 9.534836298463e-05
243 KSP preconditioned resid norm 7.523666116103e+00 true resid norm 1.151464810154e+04 ||r(i)||/||b|| 9.535940456761e-05
244 KSP preconditioned resid norm 7.521362246446e+00 true resid norm 1.151314695941e+04 ||r(i)||/||b|| 9.534697274868e-05
245 KSP preconditioned resid norm 7.518205414905e+00 true resid norm 1.151534358635e+04 ||r(i)||/||b|| 9.536516427613e-05
246 KSP preconditioned resid norm 7.516549936596e+00 true resid norm 1.151320650235e+04 ||r(i)||/||b|| 9.534746585796e-05
247 KSP preconditioned resid norm 7.516202850995e+00 true resid norm 1.151208151056e+04 ||r(i)||/||b|| 9.533814915573e-05
248 KSP preconditioned resid norm 7.513655287296e+00 true resid norm 1.151446976502e+04 ||r(i)||/||b|| 9.535792766056e-05
249 KSP preconditioned resid norm 7.511227863172e+00 true resid norm 1.151260351114e+04 ||r(i)||/||b|| 9.534247214188e-05
250 KSP preconditioned resid norm 7.507635360131e+00 true resid norm 1.151686193911e+04 ||r(i)||/||b|| 9.537773862612e-05
251 KSP preconditioned resid norm 7.503895729423e+00 true resid norm 1.151467180351e+04 ||r(i)||/||b|| 9.535960085718e-05
252 KSP preconditioned resid norm 7.498587983077e+00 true resid norm 1.151874997385e+04 ||r(i)||/||b|| 9.539337452457e-05
253 KSP preconditioned resid norm 7.495299884163e+00 true resid norm 1.151469993613e+04 ||r(i)||/||b|| 9.535983383957e-05
254 KSP preconditioned resid norm 7.492111506603e+00 true resid norm 1.151960062842e+04 ||r(i)||/||b|| 9.540041928292e-05
255 KSP preconditioned resid norm 7.490380421818e+00 true resid norm 1.151781681283e+04 ||r(i)||/||b|| 9.538564648300e-05
256 KSP preconditioned resid norm 7.489690464426e+00 true resid norm 1.151716850068e+04 ||r(i)||/||b|| 9.538027743831e-05
257 KSP preconditioned resid norm 7.488947832655e+00 true resid norm 1.151677725024e+04 ||r(i)||/||b|| 9.537703726903e-05
258 KSP preconditioned resid norm 7.487838742896e+00 true resid norm 1.151785720207e+04 ||r(i)||/||b|| 9.538598096950e-05
259 KSP preconditioned resid norm 7.487806277911e+00 true resid norm 1.151741567290e+04 ||r(i)||/||b|| 9.538232441319e-05
260 KSP preconditioned resid norm 7.486168946471e+00 true resid norm 1.151354254350e+04 ||r(i)||/||b|| 9.535024880742e-05
261 KSP preconditioned resid norm 7.481042895213e+00 true resid norm 1.151576435082e+04 ||r(i)||/||b|| 9.536864886805e-05
262 KSP preconditioned resid norm 7.472867005967e+00 true resid norm 1.152193595639e+04 ||r(i)||/||b|| 9.541975947317e-05
263 KSP preconditioned resid norm 7.471016982871e+00 true resid norm 1.151347938065e+04 ||r(i)||/||b|| 9.534972571967e-05
264 KSP preconditioned resid norm 7.470790907664e+00 true resid norm 1.151638243154e+04 ||r(i)||/||b|| 9.537376754892e-05
265 KSP preconditioned resid norm 7.470011053805e+00 true resid norm 1.151379675265e+04 ||r(i)||/||b|| 9.535235405916e-05
266 KSP preconditioned resid norm 7.465478892536e+00 true resid norm 1.152900065438e+04 ||r(i)||/||b|| 9.547826628886e-05
267 KSP preconditioned resid norm 7.459881447884e+00 true resid norm 1.151534492063e+04 ||r(i)||/||b|| 9.536517532611e-05
268 KSP preconditioned resid norm 7.458798129836e+00 true resid norm 1.152304717885e+04 ||r(i)||/||b|| 9.542896214364e-05
269 KSP preconditioned resid norm 7.458715843434e+00 true resid norm 1.152447023134e+04 ||r(i)||/||b|| 9.544074725746e-05
270 KSP preconditioned resid norm 7.456860177283e+00 true resid norm 1.151560384586e+04 ||r(i)||/||b|| 9.536731963440e-05
271 KSP preconditioned resid norm 7.456859898735e+00 true resid norm 1.151567769245e+04 ||r(i)||/||b|| 9.536793120039e-05
272 KSP preconditioned resid norm 7.456857936504e+00 true resid norm 1.151575132593e+04 ||r(i)||/||b|| 9.536854100142e-05
273 KSP preconditioned resid norm 7.456829147335e+00 true resid norm 1.151543909649e+04 ||r(i)||/||b|| 9.536595525039e-05
274 KSP preconditioned resid norm 7.456783163166e+00 true resid norm 1.151551531156e+04 ||r(i)||/||b|| 9.536658643114e-05
275 KSP preconditioned resid norm 7.456700455215e+00 true resid norm 1.151586592051e+04 ||r(i)||/||b|| 9.536949002491e-05
276 KSP preconditioned resid norm 7.456479138433e+00 true resid norm 1.151509029651e+04 ||r(i)||/||b|| 9.536306663776e-05
277 KSP preconditioned resid norm 7.456305936398e+00 true resid norm 1.151470483077e+04 ||r(i)||/||b|| 9.535987437487e-05
278 KSP preconditioned resid norm 7.455256261572e+00 true resid norm 1.151620313892e+04 ||r(i)||/||b|| 9.537228272391e-05
279 KSP preconditioned resid norm 7.453507788413e+00 true resid norm 1.151413727637e+04 ||r(i)||/||b|| 9.535517413140e-05
280 KSP preconditioned resid norm 7.452204417472e+00 true resid norm 1.151632008031e+04 ||r(i)||/||b|| 9.537325118263e-05
281 KSP preconditioned resid norm 7.452037828402e+00 true resid norm 1.151577459500e+04 ||r(i)||/||b|| 9.536873370595e-05
282 KSP preconditioned resid norm 7.452015153197e+00 true resid norm 1.151596851229e+04 ||r(i)||/||b|| 9.537033964627e-05
283 KSP preconditioned resid norm 7.451878422001e+00 true resid norm 1.151549932913e+04 ||r(i)||/||b|| 9.536645407142e-05
284 KSP preconditioned resid norm 7.451070928033e+00 true resid norm 1.151791346792e+04 ||r(i)||/||b|| 9.538644693929e-05
285 KSP preconditioned resid norm 7.450673893602e+00 true resid norm 1.151639888981e+04 ||r(i)||/||b|| 9.537390384935e-05
286 KSP preconditioned resid norm 7.449511278794e+00 true resid norm 1.151318687556e+04 ||r(i)||/||b|| 9.534730331722e-05
287 KSP preconditioned resid norm 7.444500008302e+00 true resid norm 1.152240825109e+04 ||r(i)||/||b|| 9.542367081641e-05
288 KSP preconditioned resid norm 7.437071699877e+00 true resid norm 1.151554420260e+04 ||r(i)||/||b|| 9.536682569435e-05
289 KSP preconditioned resid norm 7.434527431063e+00 true resid norm 1.152375910019e+04 ||r(i)||/||b|| 9.543485797255e-05
290 KSP preconditioned resid norm 7.433360195137e+00 true resid norm 1.151861718993e+04 ||r(i)||/||b|| 9.539227486477e-05
291 KSP preconditioned resid norm 7.433189042734e+00 true resid norm 1.152040892846e+04 ||r(i)||/||b|| 9.540711327914e-05
292 KSP preconditioned resid norm 7.431335094193e+00 true resid norm 1.151587717303e+04 ||r(i)||/||b|| 9.536958321343e-05
293 KSP preconditioned resid norm 7.429341604275e+00 true resid norm 1.151720927738e+04 ||r(i)||/||b|| 9.538061513357e-05
294 KSP preconditioned resid norm 7.427043767781e+00 true resid norm 1.152071915901e+04 ||r(i)||/||b|| 9.540968247622e-05
295 KSP preconditioned resid norm 7.426904088176e+00 true resid norm 1.152115380140e+04 ||r(i)||/||b|| 9.541328199913e-05
296 KSP preconditioned resid norm 7.425749447110e+00 true resid norm 1.152012454578e+04 ||r(i)||/||b|| 9.540475814306e-05
297 KSP preconditioned resid norm 7.425448719548e+00 true resid norm 1.151629651301e+04 ||r(i)||/||b|| 9.537305600836e-05
298 KSP preconditioned resid norm 7.424814666339e+00 true resid norm 1.151474547648e+04 ||r(i)||/||b|| 9.536021098529e-05
299 KSP preconditioned resid norm 7.420124821016e+00 true resid norm 1.152232309523e+04 ||r(i)||/||b|| 9.542296559189e-05
300 KSP preconditioned resid norm 7.414233577746e+00 true resid norm 1.150674356341e+04 ||r(i)||/||b|| 9.529394255407e-05
301 KSP preconditioned resid norm 7.414230345874e+00 true resid norm 1.150700923010e+04 ||r(i)||/||b|| 9.529614269231e-05
302 KSP preconditioned resid norm 7.414178779969e+00 true resid norm 1.150662986226e+04 ||r(i)||/||b|| 9.529300092968e-05
303 KSP preconditioned resid norm 7.414007378614e+00 true resid norm 1.150770400491e+04 ||r(i)||/||b|| 9.530189652098e-05
304 KSP preconditioned resid norm 7.413530418521e+00 true resid norm 1.150704704643e+04 ||r(i)||/||b|| 9.529645587103e-05
305 KSP preconditioned resid norm 7.412394277046e+00 true resid norm 1.150844355299e+04 ||r(i)||/||b|| 9.530802114276e-05
306 KSP preconditioned resid norm 7.410468280972e+00 true resid norm 1.150642081006e+04 ||r(i)||/||b|| 9.529126964849e-05
307 KSP preconditioned resid norm 7.410212108670e+00 true resid norm 1.150697250072e+04 ||r(i)||/||b|| 9.529583851521e-05
308 KSP preconditioned resid norm 7.410016914505e+00 true resid norm 1.150690555949e+04 ||r(i)||/||b|| 9.529528413655e-05
309 KSP preconditioned resid norm 7.409768146449e+00 true resid norm 1.150622196307e+04 ||r(i)||/||b|| 9.528962288257e-05
310 KSP preconditioned resid norm 7.409086360700e+00 true resid norm 1.150762737497e+04 ||r(i)||/||b|| 9.530126190445e-05
311 KSP preconditioned resid norm 7.408658172901e+00 true resid norm 1.150679417212e+04 ||r(i)||/||b|| 9.529436167383e-05
312 KSP preconditioned resid norm 7.407518366502e+00 true resid norm 1.150790211537e+04 ||r(i)||/||b|| 9.530353718727e-05
313 KSP preconditioned resid norm 7.405341061701e+00 true resid norm 1.150577864012e+04 ||r(i)||/||b|| 9.528595147096e-05
314 KSP preconditioned resid norm 7.402900678275e+00 true resid norm 1.150927680771e+04 ||r(i)||/||b|| 9.531492180292e-05
315 KSP preconditioned resid norm 7.402406116678e+00 true resid norm 1.150891887635e+04 ||r(i)||/||b|| 9.531195756809e-05
316 KSP preconditioned resid norm 7.401042599925e+00 true resid norm 1.150559844920e+04 ||r(i)||/||b|| 9.528445920661e-05
317 KSP preconditioned resid norm 7.399996725303e+00 true resid norm 1.150763222626e+04 ||r(i)||/||b|| 9.530130208079e-05
318 KSP preconditioned resid norm 7.398318173323e+00 true resid norm 1.150811697685e+04 ||r(i)||/||b|| 9.530531657844e-05
319 KSP preconditioned resid norm 7.396345546713e+00 true resid norm 1.150714484769e+04 ||r(i)||/||b|| 9.529726581936e-05
320 KSP preconditioned resid norm 7.395222656951e+00 true resid norm 1.150382509162e+04 ||r(i)||/||b|| 9.526977301548e-05
321 KSP preconditioned resid norm 7.394771772712e+00 true resid norm 1.150442364100e+04 ||r(i)||/||b|| 9.527472994613e-05
322 KSP preconditioned resid norm 7.392232703688e+00 true resid norm 1.150947403396e+04 ||r(i)||/||b|| 9.531655514660e-05
323 KSP preconditioned resid norm 7.391009768685e+00 true resid norm 1.150360222158e+04 ||r(i)||/||b|| 9.526792730080e-05
324 KSP preconditioned resid norm 7.390175315706e+00 true resid norm 1.151000606173e+04 ||r(i)||/||b|| 9.532096117373e-05
325 KSP preconditioned resid norm 7.389008610610e+00 true resid norm 1.150573775532e+04 ||r(i)||/||b|| 9.528561288049e-05
326 KSP preconditioned resid norm 7.388226519666e+00 true resid norm 1.151116858630e+04 ||r(i)||/||b|| 9.533058870637e-05
327 KSP preconditioned resid norm 7.384753342106e+00 true resid norm 1.150148608714e+04 ||r(i)||/||b|| 9.525040237794e-05
328 KSP preconditioned resid norm 7.380571325986e+00 true resid norm 1.151513295793e+04 ||r(i)||/||b|| 9.536341994142e-05
329 KSP preconditioned resid norm 7.378932379696e+00 true resid norm 1.150912121696e+04 ||r(i)||/||b|| 9.531363326674e-05
330 KSP preconditioned resid norm 7.377992547681e+00 true resid norm 1.150422708218e+04 ||r(i)||/||b|| 9.527310212978e-05
331 KSP preconditioned resid norm 7.377992154997e+00 true resid norm 1.150427220322e+04 ||r(i)||/||b|| 9.527347580301e-05
332 KSP preconditioned resid norm 7.377978730871e+00 true resid norm 1.150438054397e+04 ||r(i)||/||b|| 9.527437303492e-05
333 KSP preconditioned resid norm 7.377912017937e+00 true resid norm 1.150375879913e+04 ||r(i)||/||b|| 9.526922400932e-05
334 KSP preconditioned resid norm 7.377787020968e+00 true resid norm 1.150405916523e+04 ||r(i)||/||b|| 9.527171151327e-05
335 KSP preconditioned resid norm 7.377745210209e+00 true resid norm 1.150422685161e+04 ||r(i)||/||b|| 9.527310022035e-05
336 KSP preconditioned resid norm 7.377314501613e+00 true resid norm 1.150392642939e+04 ||r(i)||/||b|| 9.527061225166e-05
337 KSP preconditioned resid norm 7.376986910832e+00 true resid norm 1.150337481472e+04 ||r(i)||/||b|| 9.526604401417e-05
338 KSP preconditioned resid norm 7.376304558304e+00 true resid norm 1.150453211393e+04 ||r(i)||/||b|| 9.527562827268e-05
339 KSP preconditioned resid norm 7.374724314826e+00 true resid norm 1.150223364827e+04 ||r(i)||/||b|| 9.525659336034e-05
340 KSP preconditioned resid norm 7.373358281210e+00 true resid norm 1.150429261714e+04 ||r(i)||/||b|| 9.527364486237e-05
341 KSP preconditioned resid norm 7.373286115508e+00 true resid norm 1.150394932179e+04 ||r(i)||/||b|| 9.527080183672e-05
342 KSP preconditioned resid norm 7.373282226063e+00 true resid norm 1.150399014783e+04 ||r(i)||/||b|| 9.527113994057e-05
343 KSP preconditioned resid norm 7.373277096582e+00 true resid norm 1.150418457072e+04 ||r(i)||/||b|| 9.527275006801e-05
344 KSP preconditioned resid norm 7.372877665390e+00 true resid norm 1.150565861867e+04 ||r(i)||/||b|| 9.528495750450e-05
345 KSP preconditioned resid norm 7.372505539246e+00 true resid norm 1.150387471013e+04 ||r(i)||/||b|| 9.527018393481e-05
346 KSP preconditioned resid norm 7.371190511526e+00 true resid norm 1.150090387703e+04 ||r(i)||/||b|| 9.524558076209e-05
347 KSP preconditioned resid norm 7.366496578039e+00 true resid norm 1.150954939638e+04 ||r(i)||/||b|| 9.531717926602e-05
348 KSP preconditioned resid norm 7.363745923356e+00 true resid norm 1.150518400940e+04 ||r(i)||/||b|| 9.528102699290e-05
349 KSP preconditioned resid norm 7.362506237470e+00 true resid norm 1.151116310456e+04 ||r(i)||/||b|| 9.533054330896e-05
350 KSP preconditioned resid norm 7.360779600462e+00 true resid norm 1.150487605442e+04 ||r(i)||/||b|| 9.527847664112e-05
351 KSP preconditioned resid norm 7.360690474954e+00 true resid norm 1.150490223255e+04 ||r(i)||/||b|| 9.527869343726e-05
352 KSP preconditioned resid norm 7.360331073875e+00 true resid norm 1.150599352153e+04 ||r(i)||/||b|| 9.528773102714e-05
353 KSP preconditioned resid norm 7.359221060598e+00 true resid norm 1.150934934958e+04 ||r(i)||/||b|| 9.531552256378e-05
354 KSP preconditioned resid norm 7.357477327247e+00 true resid norm 1.151032128654e+04 ||r(i)||/||b|| 9.532357173116e-05
355 KSP preconditioned resid norm 7.357410777472e+00 true resid norm 1.150817512419e+04 ||r(i)||/||b|| 9.530579812988e-05
356 KSP preconditioned resid norm 7.356087498483e+00 true resid norm 1.150966848660e+04 ||r(i)||/||b|| 9.531816552046e-05
357 KSP preconditioned resid norm 7.355234070093e+00 true resid norm 1.150551057452e+04 ||r(i)||/||b|| 9.528373146597e-05
358 KSP preconditioned resid norm 7.355089103223e+00 true resid norm 1.150701891537e+04 ||r(i)||/||b|| 9.529622290154e-05
359 KSP preconditioned resid norm 7.353267926541e+00 true resid norm 1.151640543987e+04 ||r(i)||/||b|| 9.537395809410e-05
360 KSP preconditioned resid norm 7.349841981396e+00 true resid norm 1.150489535855e+04 ||r(i)||/||b|| 9.527863650970e-05
361 KSP preconditioned resid norm 7.349838451554e+00 true resid norm 1.150508897798e+04 ||r(i)||/||b|| 9.528023998327e-05
362 KSP preconditioned resid norm 7.349764038137e+00 true resid norm 1.150466920231e+04 ||r(i)||/||b|| 9.527676358017e-05
363 KSP preconditioned resid norm 7.349600743429e+00 true resid norm 1.150561613284e+04 ||r(i)||/||b|| 9.528460565496e-05
364 KSP preconditioned resid norm 7.349481599947e+00 true resid norm 1.150557226839e+04 ||r(i)||/||b|| 9.528424238833e-05
365 KSP preconditioned resid norm 7.349468897220e+00 true resid norm 1.150552440335e+04 ||r(i)||/||b|| 9.528384599045e-05
366 KSP preconditioned resid norm 7.348999314421e+00 true resid norm 1.150511741204e+04 ||r(i)||/||b|| 9.528047546201e-05
367 KSP preconditioned resid norm 7.348938040427e+00 true resid norm 1.150546158765e+04 ||r(i)||/||b|| 9.528332577760e-05
368 KSP preconditioned resid norm 7.348934771670e+00 true resid norm 1.150547265568e+04 ||r(i)||/||b|| 9.528341743832e-05
369 KSP preconditioned resid norm 7.348679094653e+00 true resid norm 1.150446894991e+04 ||r(i)||/||b|| 9.527510517518e-05
370 KSP preconditioned resid norm 7.348010960047e+00 true resid norm 1.150571449410e+04 ||r(i)||/||b|| 9.528542024096e-05
371 KSP preconditioned resid norm 7.347754646434e+00 true resid norm 1.150546036262e+04 ||r(i)||/||b|| 9.528331563247e-05
372 KSP preconditioned resid norm 7.347408523435e+00 true resid norm 1.150562975132e+04 ||r(i)||/||b|| 9.528471843737e-05
373 KSP preconditioned resid norm 7.346346378204e+00 true resid norm 1.150501069125e+04 ||r(i)||/||b|| 9.527959164589e-05
374 KSP preconditioned resid norm 7.345495614875e+00 true resid norm 1.150637510158e+04 ||r(i)||/||b|| 9.529089111039e-05
375 KSP preconditioned resid norm 7.344969287266e+00 true resid norm 1.150632611511e+04 ||r(i)||/||b|| 9.529048542530e-05
376 KSP preconditioned resid norm 7.344447164890e+00 true resid norm 1.150418490915e+04 ||r(i)||/||b|| 9.527275287075e-05
377 KSP preconditioned resid norm 7.344155362015e+00 true resid norm 1.150627287830e+04 ||r(i)||/||b|| 9.529004454073e-05
378 KSP preconditioned resid norm 7.342944200225e+00 true resid norm 1.150607962147e+04 ||r(i)||/||b|| 9.528844407009e-05
379 KSP preconditioned resid norm 7.341878464455e+00 true resid norm 1.150853991405e+04 ||r(i)||/||b|| 9.530881916391e-05
380 KSP preconditioned resid norm 7.341054785142e+00 true resid norm 1.150425495558e+04 ||r(i)||/||b|| 9.527333296545e-05
381 KSP preconditioned resid norm 7.340870555516e+00 true resid norm 1.150265534393e+04 ||r(i)||/||b|| 9.526008566395e-05
382 KSP preconditioned resid norm 7.340139194784e+00 true resid norm 1.150695063590e+04 ||r(i)||/||b|| 9.529565744013e-05
383 KSP preconditioned resid norm 7.338208026990e+00 true resid norm 1.150342847393e+04 ||r(i)||/||b|| 9.526648839693e-05
384 KSP preconditioned resid norm 7.337144180817e+00 true resid norm 1.151087476105e+04 ||r(i)||/||b|| 9.532815537094e-05
385 KSP preconditioned resid norm 7.335336605900e+00 true resid norm 1.151007234871e+04 ||r(i)||/||b|| 9.532151013417e-05
386 KSP preconditioned resid norm 7.334438177357e+00 true resid norm 1.150314458551e+04 ||r(i)||/||b|| 9.526413735409e-05
387 KSP preconditioned resid norm 7.334029574597e+00 true resid norm 1.150120944010e+04 ||r(i)||/||b|| 9.524811130516e-05
388 KSP preconditioned resid norm 7.331716633047e+00 true resid norm 1.151234188247e+04 ||r(i)||/||b|| 9.534030544484e-05
389 KSP preconditioned resid norm 7.331232813679e+00 true resid norm 1.151591245422e+04 ||r(i)||/||b|| 9.536987539722e-05
390 KSP preconditioned resid norm 7.330101228373e+00 true resid norm 1.150910417014e+04 ||r(i)||/||b|| 9.531349209220e-05
391 KSP preconditioned resid norm 7.330100355703e+00 true resid norm 1.150923625970e+04 ||r(i)||/||b|| 9.531458600165e-05
392 KSP preconditioned resid norm 7.330072929542e+00 true resid norm 1.150896542934e+04 ||r(i)||/||b|| 9.531234310010e-05
393 KSP preconditioned resid norm 7.330059015451e+00 true resid norm 1.150915454282e+04 ||r(i)||/||b|| 9.531390925724e-05
394 KSP preconditioned resid norm 7.329902347930e+00 true resid norm 1.150952050378e+04 ||r(i)||/||b|| 9.531693998984e-05
395 KSP preconditioned resid norm 7.329849041483e+00 true resid norm 1.150925620155e+04 ||r(i)||/||b|| 9.531475115148e-05
396 KSP preconditioned resid norm 7.329683091355e+00 true resid norm 1.150878630619e+04 ||r(i)||/||b|| 9.531085967857e-05
397 KSP preconditioned resid norm 7.329667835532e+00 true resid norm 1.150877950134e+04 ||r(i)||/||b|| 9.531080332373e-05
398 KSP preconditioned resid norm 7.329465199412e+00 true resid norm 1.150944783952e+04 ||r(i)||/||b|| 9.531633821548e-05
399 KSP preconditioned resid norm 7.328730669231e+00 true resid norm 1.150821144112e+04 ||r(i)||/||b|| 9.530609889126e-05
400 KSP preconditioned resid norm 7.327877202444e+00 true resid norm 1.150979105965e+04 ||r(i)||/||b|| 9.531918061819e-05
401 KSP preconditioned resid norm 7.327732985399e+00 true resid norm 1.150928547680e+04 ||r(i)||/||b|| 9.531499359669e-05
402 KSP preconditioned resid norm 7.327644449534e+00 true resid norm 1.150953709192e+04 ||r(i)||/||b|| 9.531707736580e-05
403 KSP preconditioned resid norm 7.327484949795e+00 true resid norm 1.151031378915e+04 ||r(i)||/||b|| 9.532350964097e-05
404 KSP preconditioned resid norm 7.327339737017e+00 true resid norm 1.151000618274e+04 ||r(i)||/||b|| 9.532096217588e-05
405 KSP preconditioned resid norm 7.327158977127e+00 true resid norm 1.150931698546e+04 ||r(i)||/||b|| 9.531525453796e-05
406 KSP preconditioned resid norm 7.325645671357e+00 true resid norm 1.150829246285e+04 ||r(i)||/||b|| 9.530676987868e-05
407 KSP preconditioned resid norm 7.324059901064e+00 true resid norm 1.151300767394e+04 ||r(i)||/||b|| 9.534581924588e-05
408 KSP preconditioned resid norm 7.323274786172e+00 true resid norm 1.151051174592e+04 ||r(i)||/||b|| 9.532514903446e-05
409 KSP preconditioned resid norm 7.323179334968e+00 true resid norm 1.151166564705e+04 ||r(i)||/||b|| 9.533470515153e-05
410 KSP preconditioned resid norm 7.322906730174e+00 true resid norm 1.150926656444e+04 ||r(i)||/||b|| 9.531483697253e-05
411 KSP preconditioned resid norm 7.322545970407e+00 true resid norm 1.150711802390e+04 ||r(i)||/||b|| 9.529704367615e-05
412 KSP preconditioned resid norm 7.321914300120e+00 true resid norm 1.151188345928e+04 ||r(i)||/||b|| 9.533650897954e-05
413 KSP preconditioned resid norm 7.321460471935e+00 true resid norm 1.151071280627e+04 ||r(i)||/||b|| 9.532681413057e-05
414 KSP preconditioned resid norm 7.319956584093e+00 true resid norm 1.151473744959e+04 ||r(i)||/||b|| 9.536014451001e-05
415 KSP preconditioned resid norm 7.319881552090e+00 true resid norm 1.151642991169e+04 ||r(i)||/||b|| 9.537416075933e-05
416 KSP preconditioned resid norm 7.319463034440e+00 true resid norm 1.151262225603e+04 ||r(i)||/||b|| 9.534262737911e-05
417 KSP preconditioned resid norm 7.317915924013e+00 true resid norm 1.150561631672e+04 ||r(i)||/||b|| 9.528460717777e-05
418 KSP preconditioned resid norm 7.317129591855e+00 true resid norm 1.150406742867e+04 ||r(i)||/||b|| 9.527177994758e-05
419 KSP preconditioned resid norm 7.313625195805e+00 true resid norm 1.151550678463e+04 ||r(i)||/||b|| 9.536651581470e-05
420 KSP preconditioned resid norm 7.313609219076e+00 true resid norm 1.151530610332e+04 ||r(i)||/||b|| 9.536485385772e-05
421 KSP preconditioned resid norm 7.313609207916e+00 true resid norm 1.151531533423e+04 ||r(i)||/||b|| 9.536493030413e-05
422 KSP preconditioned resid norm 7.313575906713e+00 true resid norm 1.151508784950e+04 ||r(i)||/||b|| 9.536304637267e-05
423 KSP preconditioned resid norm 7.313394643102e+00 true resid norm 1.151623139038e+04 ||r(i)||/||b|| 9.537251669052e-05
424 KSP preconditioned resid norm 7.313249640952e+00 true resid norm 1.151608304576e+04 ||r(i)||/||b|| 9.537128816357e-05
425 KSP preconditioned resid norm 7.313011498049e+00 true resid norm 1.151580975976e+04 ||r(i)||/||b|| 9.536902492548e-05
426 KSP preconditioned resid norm 7.312979454005e+00 true resid norm 1.151607462460e+04 ||r(i)||/||b|| 9.537121842312e-05
427 KSP preconditioned resid norm 7.312945369860e+00 true resid norm 1.151609939701e+04 ||r(i)||/||b|| 9.537142357767e-05
428 KSP preconditioned resid norm 7.312802253270e+00 true resid norm 1.151646170913e+04 ||r(i)||/||b|| 9.537442409213e-05
429 KSP preconditioned resid norm 7.312610358472e+00 true resid norm 1.151604671656e+04 ||r(i)||/||b|| 9.537098730064e-05
430 KSP preconditioned resid norm 7.312486025107e+00 true resid norm 1.151636540943e+04 ||r(i)||/||b|| 9.537362657905e-05
431 KSP preconditioned resid norm 7.312324251460e+00 true resid norm 1.151617397288e+04 ||r(i)||/||b|| 9.537204118324e-05
432 KSP preconditioned resid norm 7.312266649944e+00 true resid norm 1.151635589497e+04 ||r(i)||/||b|| 9.537354778442e-05
433 KSP preconditioned resid norm 7.311449928669e+00 true resid norm 1.151572634971e+04 ||r(i)||/||b|| 9.536833415903e-05
434 KSP preconditioned resid norm 7.310875084657e+00 true resid norm 1.151653745320e+04 ||r(i)||/||b|| 9.537505137221e-05
435 KSP preconditioned resid norm 7.310426158109e+00 true resid norm 1.151575992226e+04 ||r(i)||/||b|| 9.536861219259e-05
436 KSP preconditioned resid norm 7.310353073750e+00 true resid norm 1.151484746064e+04 ||r(i)||/||b|| 9.536105557462e-05
437 KSP preconditioned resid norm 7.310079806080e+00 true resid norm 1.151528373233e+04 ||r(i)||/||b|| 9.536466859068e-05
438 KSP preconditioned resid norm 7.309461777240e+00 true resid norm 1.151549243469e+04 ||r(i)||/||b|| 9.536639697459e-05
439 KSP preconditioned resid norm 7.309189588202e+00 true resid norm 1.151706770693e+04 ||r(i)||/||b|| 9.537944270742e-05
440 KSP preconditioned resid norm 7.309109669368e+00 true resid norm 1.151657571929e+04 ||r(i)||/||b|| 9.537536827567e-05
441 KSP preconditioned resid norm 7.308913386706e+00 true resid norm 1.151713161889e+04 ||r(i)||/||b|| 9.537997199902e-05
442 KSP preconditioned resid norm 7.308102556196e+00 true resid norm 1.151717393422e+04 ||r(i)||/||b|| 9.538032243661e-05
443 KSP preconditioned resid norm 7.305936734783e+00 true resid norm 1.151305727556e+04 ||r(i)||/||b|| 9.534623002528e-05
444 KSP preconditioned resid norm 7.305002874256e+00 true resid norm 1.151879231458e+04 ||r(i)||/||b|| 9.539372517247e-05
445 KSP preconditioned resid norm 7.304327762931e+00 true resid norm 1.151884260277e+04 ||r(i)||/||b|| 9.539414163785e-05
446 KSP preconditioned resid norm 7.304286445822e+00 true resid norm 1.151944496969e+04 ||r(i)||/||b|| 9.539913018375e-05
447 KSP preconditioned resid norm 7.304194201118e+00 true resid norm 1.151775502230e+04 ||r(i)||/||b|| 9.538513476025e-05
448 KSP preconditioned resid norm 7.303691979440e+00 true resid norm 1.152227742882e+04 ||r(i)||/||b|| 9.542258740222e-05
449 KSP preconditioned resid norm 7.303259001054e+00 true resid norm 1.151826256980e+04 ||r(i)||/||b|| 9.538933805215e-05
450 KSP preconditioned resid norm 7.302866391503e+00 true resid norm 1.151563081631e+04 ||r(i)||/||b|| 9.536754299214e-05
451 KSP preconditioned resid norm 7.302865897058e+00 true resid norm 1.151567510399e+04 ||r(i)||/||b|| 9.536790976389e-05
452 KSP preconditioned resid norm 7.302847110278e+00 true resid norm 1.151571272688e+04 ||r(i)||/||b|| 9.536822134056e-05
453 KSP preconditioned resid norm 7.302766300634e+00 true resid norm 1.151510210265e+04 ||r(i)||/||b|| 9.536316441113e-05
454 KSP preconditioned resid norm 7.302639201882e+00 true resid norm 1.151548946410e+04 ||r(i)||/||b|| 9.536637237345e-05
455 KSP preconditioned resid norm 7.302527538409e+00 true resid norm 1.151513798280e+04 ||r(i)||/||b|| 9.536346155524e-05
456 KSP preconditioned resid norm 7.302479443095e+00 true resid norm 1.151485930219e+04 ||r(i)||/||b|| 9.536115364127e-05
457 KSP preconditioned resid norm 7.302416077608e+00 true resid norm 1.151438701995e+04 ||r(i)||/||b|| 9.535724240120e-05
458 KSP preconditioned resid norm 7.301627186130e+00 true resid norm 1.151550594551e+04 ||r(i)||/||b|| 9.536650886548e-05
459 KSP preconditioned resid norm 7.300563856431e+00 true resid norm 1.151442405780e+04 ||r(i)||/||b|| 9.535754913287e-05
460 KSP preconditioned resid norm 7.299994726214e+00 true resid norm 1.151571972155e+04 ||r(i)||/||b|| 9.536827926746e-05
461 KSP preconditioned resid norm 7.299887424433e+00 true resid norm 1.151512555466e+04 ||r(i)||/||b|| 9.536335863067e-05
462 KSP preconditioned resid norm 7.299882544968e+00 true resid norm 1.151511060647e+04 ||r(i)||/||b|| 9.536323483617e-05
463 KSP preconditioned resid norm 7.299527885469e+00 true resid norm 1.151654495292e+04 ||r(i)||/||b|| 9.537511348174e-05
464 KSP preconditioned resid norm 7.299375859990e+00 true resid norm 1.151555523426e+04 ||r(i)||/||b|| 9.536691705386e-05
465 KSP preconditioned resid norm 7.299365758092e+00 true resid norm 1.151577080735e+04 ||r(i)||/||b|| 9.536870233831e-05
466 KSP preconditioned resid norm 7.299022217241e+00 true resid norm 1.151527315612e+04 ||r(i)||/||b|| 9.536458100303e-05
467 KSP preconditioned resid norm 7.298439415516e+00 true resid norm 1.151781693663e+04 ||r(i)||/||b|| 9.538564750825e-05
468 KSP preconditioned resid norm 7.297218500998e+00 true resid norm 1.151399070905e+04 ||r(i)||/||b|| 9.535396032333e-05
469 KSP preconditioned resid norm 7.296958770977e+00 true resid norm 1.151633442737e+04 ||r(i)||/||b|| 9.537336999890e-05
470 KSP preconditioned resid norm 7.296745203072e+00 true resid norm 1.151550607144e+04 ||r(i)||/||b|| 9.536650990835e-05
471 KSP preconditioned resid norm 7.296136345462e+00 true resid norm 1.151995207321e+04 ||r(i)||/||b|| 9.540332979883e-05
472 KSP preconditioned resid norm 7.295518965653e+00 true resid norm 1.151815230946e+04 ||r(i)||/||b|| 9.538842492306e-05
473 KSP preconditioned resid norm 7.294550122873e+00 true resid norm 1.151352671585e+04 ||r(i)||/||b|| 9.535011772960e-05
474 KSP preconditioned resid norm 7.294467494401e+00 true resid norm 1.151314608274e+04 ||r(i)||/||b|| 9.534696548847e-05
475 KSP preconditioned resid norm 7.294044930541e+00 true resid norm 1.151758202735e+04 ||r(i)||/||b|| 9.538370208982e-05
476 KSP preconditioned resid norm 7.293912059884e+00 true resid norm 1.152003324026e+04 ||r(i)||/||b|| 9.540400198968e-05
477 KSP preconditioned resid norm 7.291979071172e+00 true resid norm 1.151096378429e+04 ||r(i)||/||b|| 9.532889262347e-05
478 KSP preconditioned resid norm 7.290794794712e+00 true resid norm 1.151068390060e+04 ||r(i)||/||b|| 9.532657474610e-05
479 KSP preconditioned resid norm 7.288761483435e+00 true resid norm 1.152033630534e+04 ||r(i)||/||b|| 9.540651184542e-05
480 KSP preconditioned resid norm 7.287291467314e+00 true resid norm 1.151287030622e+04 ||r(i)||/||b|| 9.534468162494e-05
481 KSP preconditioned resid norm 7.287290207534e+00 true resid norm 1.151302646660e+04 ||r(i)||/||b|| 9.534597487866e-05
482 KSP preconditioned resid norm 7.287234297482e+00 true resid norm 1.151260539501e+04 ||r(i)||/||b|| 9.534248774333e-05
483 KSP preconditioned resid norm 7.287008093466e+00 true resid norm 1.151383160298e+04 ||r(i)||/||b|| 9.535264267477e-05
484 KSP preconditioned resid norm 7.286782487348e+00 true resid norm 1.151379199415e+04 ||r(i)||/||b|| 9.535231465131e-05
485 KSP preconditioned resid norm 7.286288277712e+00 true resid norm 1.151332915665e+04 ||r(i)||/||b|| 9.534848162852e-05
486 KSP preconditioned resid norm 7.286232808730e+00 true resid norm 1.151364944398e+04 ||r(i)||/||b|| 9.535113411161e-05
487 KSP preconditioned resid norm 7.286157845759e+00 true resid norm 1.151315768631e+04 ||r(i)||/||b|| 9.534706158433e-05
488 KSP preconditioned resid norm 7.285801329869e+00 true resid norm 1.151396046686e+04 ||r(i)||/||b|| 9.535370987048e-05
489 KSP preconditioned resid norm 7.285545164211e+00 true resid norm 1.151356765840e+04 ||r(i)||/||b|| 9.535045679828e-05
490 KSP preconditioned resid norm 7.285504936889e+00 true resid norm 1.151317352747e+04 ||r(i)||/||b|| 9.534719277405e-05
491 KSP preconditioned resid norm 7.285382544999e+00 true resid norm 1.151374827165e+04 ||r(i)||/||b|| 9.535195256024e-05
492 KSP preconditioned resid norm 7.285289746039e+00 true resid norm 1.151347615695e+04 ||r(i)||/||b|| 9.534969902233e-05
493 KSP preconditioned resid norm 7.284600626969e+00 true resid norm 1.151310144469e+04 ||r(i)||/||b|| 9.534659581517e-05
494 KSP preconditioned resid norm 7.284140392304e+00 true resid norm 1.151355776927e+04 ||r(i)||/||b|| 9.535037490073e-05
495 KSP preconditioned resid norm 7.284018682787e+00 true resid norm 1.151344794812e+04 ||r(i)||/||b|| 9.534946540885e-05
496 KSP preconditioned resid norm 7.283943356662e+00 true resid norm 1.151335471392e+04 ||r(i)||/||b|| 9.534869328294e-05
497 KSP preconditioned resid norm 7.283498746723e+00 true resid norm 1.151386532333e+04 ||r(i)||/||b|| 9.535292193227e-05
498 KSP preconditioned resid norm 7.283151422198e+00 true resid norm 1.151281083802e+04 ||r(i)||/||b|| 9.534418913469e-05
499 KSP preconditioned resid norm 7.282874986659e+00 true resid norm 1.151390183052e+04 ||r(i)||/||b|| 9.535322426928e-05
500 KSP preconditioned resid norm 7.282665039613e+00 true resid norm 1.151488350752e+04 ||r(i)||/||b|| 9.536135409947e-05
501 KSP preconditioned resid norm 7.282645456330e+00 true resid norm 1.151496780885e+04 ||r(i)||/||b|| 9.536205224716e-05
502 KSP preconditioned resid norm 7.282599155909e+00 true resid norm 1.151519364668e+04 ||r(i)||/||b|| 9.536392253977e-05
503 KSP preconditioned resid norm 7.282038998975e+00 true resid norm 1.151285843597e+04 ||r(i)||/||b|| 9.534458332063e-05
504 KSP preconditioned resid norm 7.280328166482e+00 true resid norm 1.152036054694e+04 ||r(i)||/||b|| 9.540671260400e-05
505 KSP preconditioned resid norm 7.279690974000e+00 true resid norm 1.151417435239e+04 ||r(i)||/||b|| 9.535548117920e-05
506 KSP preconditioned resid norm 7.278575861294e+00 true resid norm 1.151730773365e+04 ||r(i)||/||b|| 9.538143050641e-05
507 KSP preconditioned resid norm 7.278393128390e+00 true resid norm 1.151504820102e+04 ||r(i)||/||b|| 9.536271802085e-05
508 KSP preconditioned resid norm 7.277293254772e+00 true resid norm 1.152212843431e+04 ||r(i)||/||b|| 9.542135349323e-05
509 KSP preconditioned resid norm 7.276694016009e+00 true resid norm 1.151647515502e+04 ||r(i)||/||b|| 9.537453544529e-05
510 KSP preconditioned resid norm 7.275854185034e+00 true resid norm 1.152209239423e+04 ||r(i)||/||b|| 9.542105502464e-05
511 KSP preconditioned resid norm 7.275853758618e+00 true resid norm 1.152199923202e+04 ||r(i)||/||b|| 9.542028349497e-05
512 KSP preconditioned resid norm 7.275837931448e+00 true resid norm 1.152221857945e+04 ||r(i)||/||b|| 9.542210003678e-05
513 KSP preconditioned resid norm 7.275668873638e+00 true resid norm 1.152115606217e+04 ||r(i)||/||b|| 9.541330072191e-05
514 KSP preconditioned resid norm 7.275489233899e+00 true resid norm 1.152166508765e+04 ||r(i)||/||b|| 9.541751625377e-05
515 KSP preconditioned resid norm 7.275424137470e+00 true resid norm 1.152140479705e+04 ||r(i)||/||b|| 9.541536063802e-05
516 KSP preconditioned resid norm 7.275396705487e+00 true resid norm 1.152159520021e+04 ||r(i)||/||b|| 9.541693747584e-05
517 KSP preconditioned resid norm 7.275115213240e+00 true resid norm 1.152068967875e+04 ||r(i)||/||b|| 9.540943833333e-05
518 KSP preconditioned resid norm 7.274504814904e+00 true resid norm 1.152188182407e+04 ||r(i)||/||b|| 9.541931117240e-05
519 KSP preconditioned resid norm 7.273992218401e+00 true resid norm 1.152108837938e+04 ||r(i)||/||b|| 9.541274020184e-05
520 KSP preconditioned resid norm 7.273638498650e+00 true resid norm 1.152228105965e+04 ||r(i)||/||b|| 9.542261747115e-05
521 KSP preconditioned resid norm 7.273481479773e+00 true resid norm 1.152171931383e+04 ||r(i)||/||b|| 9.541796533189e-05
522 KSP preconditioned resid norm 7.273443665210e+00 true resid norm 1.152131555632e+04 ||r(i)||/||b|| 9.541462158441e-05
523 KSP preconditioned resid norm 7.273107174061e+00 true resid norm 1.152296872734e+04 ||r(i)||/||b|| 9.542831244170e-05
524 KSP preconditioned resid norm 7.272560750335e+00 true resid norm 1.152110544164e+04 ||r(i)||/||b|| 9.541288150425e-05
525 KSP preconditioned resid norm 7.272413188538e+00 true resid norm 1.152202988133e+04 ||r(i)||/||b|| 9.542053731946e-05
526 KSP preconditioned resid norm 7.272371364044e+00 true resid norm 1.152136050124e+04 ||r(i)||/||b|| 9.541499379907e-05
527 KSP preconditioned resid norm 7.271352150224e+00 true resid norm 1.152497859614e+04 ||r(i)||/||b|| 9.544495731787e-05
528 KSP preconditioned resid norm 7.270200787629e+00 true resid norm 1.152140103290e+04 ||r(i)||/||b|| 9.541532946497e-05
529 KSP preconditioned resid norm 7.269327671632e+00 true resid norm 1.152612323307e+04 ||r(i)||/||b|| 9.545443671273e-05
530 KSP preconditioned resid norm 7.268398267746e+00 true resid norm 1.152129444616e+04 ||r(i)||/||b|| 9.541444675905e-05
531 KSP preconditioned resid norm 7.268037737234e+00 true resid norm 1.152454898316e+04 ||r(i)||/||b|| 9.544139944642e-05
532 KSP preconditioned resid norm 7.267976776865e+00 true resid norm 1.152293929632e+04 ||r(i)||/||b|| 9.542806870659e-05
533 KSP preconditioned resid norm 7.267530209661e+00 true resid norm 1.152566137971e+04 ||r(i)||/||b|| 9.545061184020e-05
534 KSP preconditioned resid norm 7.266896921459e+00 true resid norm 1.152385057185e+04 ||r(i)||/||b|| 9.543561550184e-05
535 KSP preconditioned resid norm 7.266809989691e+00 true resid norm 1.152326670591e+04 ||r(i)||/||b|| 9.543078017310e-05
536 KSP preconditioned resid norm 7.266794672665e+00 true resid norm 1.152410511845e+04 ||r(i)||/||b|| 9.543772354821e-05
537 KSP preconditioned resid norm 7.266253729966e+00 true resid norm 1.151979653721e+04 ||r(i)||/||b|| 9.540204171597e-05
538 KSP preconditioned resid norm 7.265747858338e+00 true resid norm 1.152112709990e+04 ||r(i)||/||b|| 9.541306086867e-05
539 KSP preconditioned resid norm 7.265602463124e+00 true resid norm 1.152377350605e+04 ||r(i)||/||b|| 9.543497727573e-05
540 KSP preconditioned resid norm 7.265470322467e+00 true resid norm 1.152186963193e+04 ||r(i)||/||b|| 9.541921020228e-05
541 KSP preconditioned resid norm 7.265470250270e+00 true resid norm 1.152190395716e+04 ||r(i)||/||b|| 9.541949446918e-05
542 KSP preconditioned resid norm 7.265468010340e+00 true resid norm 1.152181324807e+04 ||r(i)||/||b|| 9.541874325518e-05
543 KSP preconditioned resid norm 7.265440006833e+00 true resid norm 1.152207032153e+04 ||r(i)||/||b|| 9.542087222793e-05
544 KSP preconditioned resid norm 7.265377761979e+00 true resid norm 1.152199690071e+04 ||r(i)||/||b|| 9.542026418802e-05
545 KSP preconditioned resid norm 7.265282670567e+00 true resid norm 1.152186191605e+04 ||r(i)||/||b|| 9.541914630268e-05
546 KSP preconditioned resid norm 7.265206072378e+00 true resid norm 1.152225933095e+04 ||r(i)||/||b|| 9.542243752339e-05
547 KSP preconditioned resid norm 7.265061658712e+00 true resid norm 1.152144362264e+04 ||r(i)||/||b|| 9.541568217502e-05
548 KSP preconditioned resid norm 7.264797877322e+00 true resid norm 1.152214345262e+04 ||r(i)||/||b|| 9.542147786849e-05
549 KSP preconditioned resid norm 7.264614796204e+00 true resid norm 1.152194201056e+04 ||r(i)||/||b|| 9.541980961126e-05
550 KSP preconditioned resid norm 7.264568050076e+00 true resid norm 1.152189336325e+04 ||r(i)||/||b|| 9.541940673492e-05
551 KSP preconditioned resid norm 7.264496641657e+00 true resid norm 1.152226288967e+04 ||r(i)||/||b|| 9.542246699518e-05
552 KSP preconditioned resid norm 7.264453655081e+00 true resid norm 1.152181454082e+04 ||r(i)||/||b|| 9.541875396119e-05
553 KSP preconditioned resid norm 7.264264957257e+00 true resid norm 1.152164903724e+04 ||r(i)||/||b|| 9.541738333113e-05
554 KSP preconditioned resid norm 7.263753586639e+00 true resid norm 1.152191161205e+04 ||r(i)||/||b|| 9.541955786376e-05
555 KSP preconditioned resid norm 7.263171483201e+00 true resid norm 1.152161125401e+04 ||r(i)||/||b|| 9.541707042657e-05
556 KSP preconditioned resid norm 7.263052047833e+00 true resid norm 1.152165490122e+04 ||r(i)||/||b|| 9.541743189411e-05
557 KSP preconditioned resid norm 7.262892944030e+00 true resid norm 1.152206717006e+04 ||r(i)||/||b|| 9.542084612879e-05
558 KSP preconditioned resid norm 7.262531124595e+00 true resid norm 1.152322923833e+04 ||r(i)||/||b|| 9.543046988261e-05
559 KSP preconditioned resid norm 7.262008845806e+00 true resid norm 1.152237326371e+04 ||r(i)||/||b|| 9.542338106590e-05
560 KSP preconditioned resid norm 7.261535819858e+00 true resid norm 1.152037988941e+04 ||r(i)||/||b|| 9.540687279008e-05
561 KSP preconditioned resid norm 7.261413309296e+00 true resid norm 1.152189156145e+04 ||r(i)||/||b|| 9.541939181324e-05
562 KSP preconditioned resid norm 7.261211653290e+00 true resid norm 1.152415509981e+04 ||r(i)||/||b|| 9.543813747252e-05
563 KSP preconditioned resid norm 7.259880913895e+00 true resid norm 1.151966814509e+04 ||r(i)||/||b|| 9.540097842723e-05
564 KSP preconditioned resid norm 7.258748521861e+00 true resid norm 1.152680043031e+04 ||r(i)||/||b|| 9.546004497148e-05
565 KSP preconditioned resid norm 7.258192078843e+00 true resid norm 1.152104214469e+04 ||r(i)||/||b|| 9.541235730588e-05
566 KSP preconditioned resid norm 7.257940057490e+00 true resid norm 1.152418813007e+04 ||r(i)||/||b|| 9.543841101506e-05
567 KSP preconditioned resid norm 7.257454264997e+00 true resid norm 1.152019944840e+04 ||r(i)||/||b|| 9.540537845466e-05
568 KSP preconditioned resid norm 7.256138700307e+00 true resid norm 1.152886220767e+04 ||r(i)||/||b|| 9.547711973222e-05
569 KSP preconditioned resid norm 7.253799442063e+00 true resid norm 1.151656094917e+04 ||r(i)||/||b|| 9.537524595582e-05
570 KSP preconditioned resid norm 7.251490764986e+00 true resid norm 1.152508476470e+04 ||r(i)||/||b|| 9.544583656059e-05
571 KSP preconditioned resid norm 7.251489563913e+00 true resid norm 1.152492143506e+04 ||r(i)||/||b|| 9.544448393420e-05
572 KSP preconditioned resid norm 7.251434458866e+00 true resid norm 1.152533480531e+04 ||r(i)||/||b|| 9.544790729030e-05
573 KSP preconditioned resid norm 7.250984833966e+00 true resid norm 1.152373433467e+04 ||r(i)||/||b|| 9.543465287508e-05
574 KSP preconditioned resid norm 7.250511804099e+00 true resid norm 1.152461089483e+04 ||r(i)||/||b|| 9.544191217245e-05
575 KSP preconditioned resid norm 7.250340683777e+00 true resid norm 1.152412641167e+04 ||r(i)||/||b|| 9.543789988960e-05
576 KSP preconditioned resid norm 7.250014491718e+00 true resid norm 1.152440548107e+04 ||r(i)||/||b|| 9.544021102331e-05
577 KSP preconditioned resid norm 7.249363058341e+00 true resid norm 1.152379552507e+04 ||r(i)||/||b|| 9.543515962790e-05
578 KSP preconditioned resid norm 7.249133923494e+00 true resid norm 1.152473378241e+04 ||r(i)||/||b|| 9.544292987499e-05
579 KSP preconditioned resid norm 7.249026206199e+00 true resid norm 1.152434883343e+04 ||r(i)||/||b|| 9.543974189172e-05
580 KSP preconditioned resid norm 7.248868128483e+00 true resid norm 1.152480627369e+04 ||r(i)||/||b|| 9.544353021682e-05
581 KSP preconditioned resid norm 7.248752715714e+00 true resid norm 1.152444031541e+04 ||r(i)||/||b|| 9.544049950650e-05
582 KSP preconditioned resid norm 7.248600063684e+00 true resid norm 1.152348169172e+04 ||r(i)||/||b|| 9.543256059390e-05
583 KSP preconditioned resid norm 7.247936178567e+00 true resid norm 1.152600183401e+04 ||r(i)||/||b|| 9.545343133755e-05
584 KSP preconditioned resid norm 7.247389109770e+00 true resid norm 1.152428867349e+04 ||r(i)||/||b|| 9.543924367277e-05
585 KSP preconditioned resid norm 7.247291941428e+00 true resid norm 1.152527366535e+04 ||r(i)||/||b|| 9.544740095522e-05
586 KSP preconditioned resid norm 7.246822954821e+00 true resid norm 1.152368643608e+04 ||r(i)||/||b|| 9.543425619934e-05
587 KSP preconditioned resid norm 7.245964187331e+00 true resid norm 1.152706085387e+04 ||r(i)||/||b|| 9.546220168835e-05
588 KSP preconditioned resid norm 7.245194412374e+00 true resid norm 1.152428565745e+04 ||r(i)||/||b|| 9.543921869521e-05
589 KSP preconditioned resid norm 7.244540957119e+00 true resid norm 1.152827454547e+04 ||r(i)||/||b|| 9.547225296452e-05
590 KSP preconditioned resid norm 7.244224082648e+00 true resid norm 1.152700764306e+04 ||r(i)||/||b|| 9.546176101907e-05
591 KSP preconditioned resid norm 7.244190331089e+00 true resid norm 1.152608622717e+04 ||r(i)||/||b|| 9.545413024568e-05
592 KSP preconditioned resid norm 7.244035266879e+00 true resid norm 1.152738682608e+04 ||r(i)||/||b|| 9.546490125110e-05
593 KSP preconditioned resid norm 7.243904122087e+00 true resid norm 1.152501097256e+04 ||r(i)||/||b|| 9.544522544558e-05
594 KSP preconditioned resid norm 7.243866526999e+00 true resid norm 1.152517242117e+04 ||r(i)||/||b|| 9.544656249412e-05
595 KSP preconditioned resid norm 7.243254736181e+00 true resid norm 1.152908323283e+04 ||r(i)||/||b|| 9.547895016834e-05
596 KSP preconditioned resid norm 7.242527558982e+00 true resid norm 1.152926869795e+04 ||r(i)||/||b|| 9.548048611134e-05
597 KSP preconditioned resid norm 7.240775409324e+00 true resid norm 1.152615337435e+04 ||r(i)||/||b|| 9.545468632996e-05
598 KSP preconditioned resid norm 7.240365240120e+00 true resid norm 1.152295012126e+04 ||r(i)||/||b|| 9.542815835408e-05
599 KSP preconditioned resid norm 7.239879140177e+00 true resid norm 1.152867934435e+04 ||r(i)||/||b|| 9.547560533619e-05
600 KSP preconditioned resid norm 7.239860181749e+00 true resid norm 1.152819731330e+04 ||r(i)||/||b|| 9.547161336060e-05
601 KSP preconditioned resid norm 7.239860177065e+00 true resid norm 1.152820441913e+04 ||r(i)||/||b|| 9.547167220806e-05
602 KSP preconditioned resid norm 7.239858943917e+00 true resid norm 1.152814093470e+04 ||r(i)||/||b|| 9.547114645713e-05
603 KSP preconditioned resid norm 7.239847688479e+00 true resid norm 1.152838053343e+04 ||r(i)||/||b|| 9.547313071157e-05
604 KSP preconditioned resid norm 7.239795894359e+00 true resid norm 1.152819136442e+04 ||r(i)||/||b|| 9.547156409452e-05
605 KSP preconditioned resid norm 7.239722259202e+00 true resid norm 1.152819193000e+04 ||r(i)||/||b|| 9.547156877847e-05
606 KSP preconditioned resid norm 7.239600914143e+00 true resid norm 1.152868688995e+04 ||r(i)||/||b|| 9.547566782562e-05
607 KSP preconditioned resid norm 7.239460031892e+00 true resid norm 1.152792156068e+04 ||r(i)||/||b|| 9.546932969502e-05
608 KSP preconditioned resid norm 7.239326898394e+00 true resid norm 1.152838733358e+04 ||r(i)||/||b|| 9.547318702750e-05
609 KSP preconditioned resid norm 7.239148142373e+00 true resid norm 1.152798506853e+04 ||r(i)||/||b|| 9.546985563996e-05
610 KSP preconditioned resid norm 7.239094877917e+00 true resid norm 1.152831360442e+04 ||r(i)||/||b|| 9.547257643407e-05
611 KSP preconditioned resid norm 7.239090800806e+00 true resid norm 1.152841706892e+04 ||r(i)||/||b|| 9.547343328298e-05
612 KSP preconditioned resid norm 7.239085383750e+00 true resid norm 1.152822976395e+04 ||r(i)||/||b|| 9.547188210307e-05
613 KSP preconditioned resid norm 7.238973165224e+00 true resid norm 1.152781565572e+04 ||r(i)||/||b|| 9.546845263531e-05
614 KSP preconditioned resid norm 7.238755402521e+00 true resid norm 1.152833674616e+04 ||r(i)||/||b|| 9.547276808407e-05
615 KSP preconditioned resid norm 7.238441015351e+00 true resid norm 1.152760651925e+04 ||r(i)||/||b|| 9.546672065631e-05
616 KSP preconditioned resid norm 7.238371186374e+00 true resid norm 1.152782740245e+04 ||r(i)||/||b|| 9.546854991671e-05
617 KSP preconditioned resid norm 7.238311087640e+00 true resid norm 1.152778088527e+04 ||r(i)||/||b|| 9.546816468126e-05
618 KSP preconditioned resid norm 7.237940632897e+00 true resid norm 1.152983896962e+04 ||r(i)||/||b|| 9.548520885807e-05
619 KSP preconditioned resid norm 7.237755395886e+00 true resid norm 1.152878954570e+04 ||r(i)||/||b|| 9.547651797675e-05
620 KSP preconditioned resid norm 7.237582247158e+00 true resid norm 1.152685630480e+04 ||r(i)||/||b|| 9.546050770018e-05
621 KSP preconditioned resid norm 7.237226198991e+00 true resid norm 1.153004449014e+04 ||r(i)||/||b|| 9.548691089143e-05
622 KSP preconditioned resid norm 7.237211596417e+00 true resid norm 1.152926453383e+04 ||r(i)||/||b|| 9.548045162591e-05
623 KSP preconditioned resid norm 7.236380115862e+00 true resid norm 1.152607342088e+04 ||r(i)||/||b|| 9.545402418943e-05
624 KSP preconditioned resid norm 7.235937552090e+00 true resid norm 1.153086800685e+04 ||r(i)||/||b|| 9.549373090559e-05
625 KSP preconditioned resid norm 7.235883413429e+00 true resid norm 1.153007449912e+04 ||r(i)||/||b|| 9.548715941297e-05
626 KSP preconditioned resid norm 7.235879326532e+00 true resid norm 1.153055080012e+04 ||r(i)||/||b|| 9.549110393474e-05
627 KSP preconditioned resid norm 7.235768071877e+00 true resid norm 1.152963190816e+04 ||r(i)||/||b|| 9.548349406338e-05
628 KSP preconditioned resid norm 7.235690891509e+00 true resid norm 1.153090154039e+04 ||r(i)||/||b|| 9.549400861600e-05
629 KSP preconditioned resid norm 7.234949298122e+00 true resid norm 1.152519145516e+04 ||r(i)||/||b|| 9.544672012552e-05
630 KSP preconditioned resid norm 7.234364564641e+00 true resid norm 1.152882160781e+04 ||r(i)||/||b|| 9.547678350146e-05
631 KSP preconditioned resid norm 7.234364364700e+00 true resid norm 1.152876604575e+04 ||r(i)||/||b|| 9.547632336018e-05
632 KSP preconditioned resid norm 7.234354801136e+00 true resid norm 1.152890565147e+04 ||r(i)||/||b|| 9.547747951524e-05
633 KSP preconditioned resid norm 7.234278664269e+00 true resid norm 1.152825365976e+04 ||r(i)||/||b|| 9.547207999798e-05
634 KSP preconditioned resid norm 7.234219066524e+00 true resid norm 1.152850619029e+04 ||r(i)||/||b|| 9.547417134810e-05
635 KSP preconditioned resid norm 7.234190634804e+00 true resid norm 1.152838349380e+04 ||r(i)||/||b|| 9.547315522810e-05
636 KSP preconditioned resid norm 7.234134188007e+00 true resid norm 1.152827900394e+04 ||r(i)||/||b|| 9.547228988764e-05
637 KSP preconditioned resid norm 7.234080394679e+00 true resid norm 1.152797416704e+04 ||r(i)||/||b|| 9.546976535845e-05
638 KSP preconditioned resid norm 7.234025063696e+00 true resid norm 1.152829459869e+04 ||r(i)||/||b|| 9.547241903673e-05
639 KSP preconditioned resid norm 7.233948805669e+00 true resid norm 1.152804182745e+04 ||r(i)||/||b|| 9.547032569316e-05
640 KSP preconditioned resid norm 7.233727120646e+00 true resid norm 1.152878506261e+04 ||r(i)||/||b|| 9.547648084972e-05
641 KSP preconditioned resid norm 7.233573051224e+00 true resid norm 1.152830344872e+04 ||r(i)||/||b|| 9.547249232891e-05
642 KSP preconditioned resid norm 7.233510544718e+00 true resid norm 1.152796018823e+04 ||r(i)||/||b|| 9.546964959194e-05
643 KSP preconditioned resid norm 7.233426010477e+00 true resid norm 1.152871376860e+04 ||r(i)||/||b|| 9.547589042312e-05
644 KSP preconditioned resid norm 7.233349551908e+00 true resid norm 1.152822186770e+04 ||r(i)||/||b|| 9.547181670968e-05
645 KSP preconditioned resid norm 7.233126226501e+00 true resid norm 1.152871089895e+04 ||r(i)||/||b|| 9.547586665790e-05
646 KSP preconditioned resid norm 7.233017491883e+00 true resid norm 1.152856655069e+04 ||r(i)||/||b|| 9.547467122724e-05
647 KSP preconditioned resid norm 7.232886332478e+00 true resid norm 1.152982645641e+04 ||r(i)||/||b|| 9.548510522906e-05
648 KSP preconditioned resid norm 7.232563212020e+00 true resid norm 1.152776037639e+04 ||r(i)||/||b|| 9.546799483546e-05
649 KSP preconditioned resid norm 7.232251790221e+00 true resid norm 1.153027489165e+04 ||r(i)||/||b|| 9.548881897846e-05
650 KSP preconditioned resid norm 7.232129020201e+00 true resid norm 1.152857468616e+04 ||r(i)||/||b|| 9.547473860170e-05
651 KSP preconditioned resid norm 7.231964184308e+00 true resid norm 1.153083289935e+04 ||r(i)||/||b|| 9.549344016020e-05
652 KSP preconditioned resid norm 7.231947601817e+00 true resid norm 1.153125446648e+04 ||r(i)||/||b|| 9.549693139936e-05
653 KSP preconditioned resid norm 7.231800183737e+00 true resid norm 1.152873432896e+04 ||r(i)||/||b|| 9.547606069527e-05
654 KSP preconditioned resid norm 7.231678093131e+00 true resid norm 1.152917975314e+04 ||r(i)||/||b|| 9.547974950837e-05
655 KSP preconditioned resid norm 7.231664588635e+00 true resid norm 1.152916305895e+04 ||r(i)||/||b|| 9.547961125421e-05
656 KSP preconditioned resid norm 7.231451602667e+00 true resid norm 1.153101668557e+04 ||r(i)||/||b|| 9.549496219934e-05
657 KSP preconditioned resid norm 7.230996541342e+00 true resid norm 1.152822012881e+04 ||r(i)||/||b|| 9.547180230896e-05
658 KSP preconditioned resid norm 7.230855615731e+00 true resid norm 1.152954237047e+04 ||r(i)||/||b|| 9.548275255044e-05
659 KSP preconditioned resid norm 7.230636061131e+00 true resid norm 1.153302847782e+04 ||r(i)||/||b|| 9.551162300472e-05
660 KSP preconditioned resid norm 7.230604378740e+00 true resid norm 1.153204752236e+04 ||r(i)||/||b|| 9.550349914995e-05
661 KSP preconditioned resid norm 7.230604371686e+00 true resid norm 1.153205953340e+04 ||r(i)||/||b|| 9.550359862023e-05
662 KSP preconditioned resid norm 7.230603278245e+00 true resid norm 1.153199472659e+04 ||r(i)||/||b|| 9.550306191791e-05
663 KSP preconditioned resid norm 7.230596130197e+00 true resid norm 1.153216730970e+04 ||r(i)||/||b|| 9.550449117759e-05
664 KSP preconditioned resid norm 7.230576064605e+00 true resid norm 1.153208134235e+04 ||r(i)||/||b|| 9.550377923267e-05
665 KSP preconditioned resid norm 7.230525259801e+00 true resid norm 1.153199468612e+04 ||r(i)||/||b|| 9.550306158277e-05
666 KSP preconditioned resid norm 7.230469072487e+00 true resid norm 1.153237183534e+04 ||r(i)||/||b|| 9.550618497174e-05
667 KSP preconditioned resid norm 7.230428382743e+00 true resid norm 1.153198396388e+04 ||r(i)||/||b|| 9.550297278573e-05
668 KSP preconditioned resid norm 7.230354094912e+00 true resid norm 1.153219824841e+04 ||r(i)||/||b|| 9.550474739879e-05
669 KSP preconditioned resid norm 7.230250574637e+00 true resid norm 1.153181509432e+04 ||r(i)||/||b|| 9.550157428008e-05
670 KSP preconditioned resid norm 7.230213492981e+00 true resid norm 1.153217879136e+04 ||r(i)||/||b|| 9.550458626385e-05
671 KSP preconditioned resid norm 7.230202459618e+00 true resid norm 1.153205557753e+04 ||r(i)||/||b|| 9.550356585945e-05
672 KSP preconditioned resid norm 7.230197672348e+00 true resid norm 1.153206634762e+04 ||r(i)||/||b|| 9.550365505275e-05
673 KSP preconditioned resid norm 7.230167494311e+00 true resid norm 1.153199136702e+04 ||r(i)||/||b|| 9.550303409540e-05
674 KSP preconditioned resid norm 7.230035653035e+00 true resid norm 1.153211098379e+04 ||r(i)||/||b|| 9.550402471045e-05
675 KSP preconditioned resid norm 7.229822206196e+00 true resid norm 1.153243087251e+04 ||r(i)||/||b|| 9.550667389242e-05
676 KSP preconditioned resid norm 7.229745194942e+00 true resid norm 1.153168873495e+04 ||r(i)||/||b|| 9.550052782560e-05
677 KSP preconditioned resid norm 7.229691345098e+00 true resid norm 1.153235166015e+04 ||r(i)||/||b|| 9.550601788944e-05
678 KSP preconditioned resid norm 7.229596198373e+00 true resid norm 1.153305937197e+04 ||r(i)||/||b|| 9.551187885684e-05
679 KSP preconditioned resid norm 7.229514439482e+00 true resid norm 1.153196171867e+04 ||r(i)||/||b|| 9.550278856039e-05
680 KSP preconditioned resid norm 7.229489404427e+00 true resid norm 1.153126730766e+04 ||r(i)||/||b|| 9.549703774455e-05
681 KSP preconditioned resid norm 7.229046547648e+00 true resid norm 1.153479391419e+04 ||r(i)||/||b|| 9.552624359573e-05
682 KSP preconditioned resid norm 7.229032181129e+00 true resid norm 1.153420061682e+04 ||r(i)||/||b|| 9.552133016000e-05
683 KSP preconditioned resid norm 7.228557338545e+00 true resid norm 1.153135776966e+04 ||r(i)||/||b|| 9.549778691229e-05
684 KSP preconditioned resid norm 7.228325130809e+00 true resid norm 1.153357453005e+04 ||r(i)||/||b|| 9.551614517637e-05
685 KSP preconditioned resid norm 7.228314507377e+00 true resid norm 1.153283953997e+04 ||r(i)||/||b|| 9.551005830200e-05
686 KSP preconditioned resid norm 7.228312559364e+00 true resid norm 1.153314606797e+04 ||r(i)||/||b|| 9.551259683617e-05
687 KSP preconditioned resid norm 7.228194670475e+00 true resid norm 1.153435339610e+04 ||r(i)||/||b|| 9.552259541280e-05
688 KSP preconditioned resid norm 7.228135675265e+00 true resid norm 1.153590870273e+04 ||r(i)||/||b|| 9.553547579899e-05
689 KSP preconditioned resid norm 7.227729566185e+00 true resid norm 1.153256414517e+04 ||r(i)||/||b|| 9.550777759974e-05
690 KSP preconditioned resid norm 7.227517726415e+00 true resid norm 1.153483860765e+04 ||r(i)||/||b|| 9.552661372793e-05
691 KSP preconditioned resid norm 7.227517672118e+00 true resid norm 1.153481262745e+04 ||r(i)||/||b|| 9.552639857101e-05
692 KSP preconditioned resid norm 7.227514639116e+00 true resid norm 1.153487003280e+04 ||r(i)||/||b|| 9.552687397761e-05
693 KSP preconditioned resid norm 7.227482802600e+00 true resid norm 1.153446502261e+04 ||r(i)||/||b|| 9.552351985597e-05
694 KSP preconditioned resid norm 7.227445155829e+00 true resid norm 1.153466094707e+04 ||r(i)||/||b|| 9.552514241876e-05
695 KSP preconditioned resid norm 7.227422673032e+00 true resid norm 1.153452492955e+04 ||r(i)||/||b|| 9.552401597966e-05
696 KSP preconditioned resid norm 7.227378993254e+00 true resid norm 1.153435516642e+04 ||r(i)||/||b|| 9.552261007386e-05
697 KSP preconditioned resid norm 7.227344014336e+00 true resid norm 1.153430484551e+04 ||r(i)||/||b|| 9.552219333751e-05
698 KSP preconditioned resid norm 7.227309569069e+00 true resid norm 1.153442412758e+04 ||r(i)||/||b|| 9.552318118079e-05
699 KSP preconditioned resid norm 7.227242671242e+00 true resid norm 1.153427138446e+04 ||r(i)||/||b|| 9.552191622739e-05
700 KSP preconditioned resid norm 7.227092400246e+00 true resid norm 1.153487147175e+04 ||r(i)||/||b|| 9.552688589434e-05
701 KSP preconditioned resid norm 7.226987517276e+00 true resid norm 1.153453711038e+04 ||r(i)||/||b|| 9.552411685611e-05
702 KSP preconditioned resid norm 7.226938107628e+00 true resid norm 1.153434935487e+04 ||r(i)||/||b|| 9.552256194504e-05
703 KSP preconditioned resid norm 7.226864995113e+00 true resid norm 1.153492928497e+04 ||r(i)||/||b|| 9.552736467880e-05
704 KSP preconditioned resid norm 7.226771215634e+00 true resid norm 1.153417451617e+04 ||r(i)||/||b|| 9.552111400550e-05
705 KSP preconditioned resid norm 7.226408258028e+00 true resid norm 1.153522795923e+04 ||r(i)||/||b|| 9.552983817163e-05
706 KSP preconditioned resid norm 7.226215056971e+00 true resid norm 1.153530554882e+04 ||r(i)||/||b|| 9.553048073557e-05
707 KSP preconditioned resid norm 7.226089581709e+00 true resid norm 1.153598659427e+04 ||r(i)||/||b|| 9.553612086351e-05
708 KSP preconditioned resid norm 7.225820927325e+00 true resid norm 1.153445508448e+04 ||r(i)||/||b|| 9.552343755258e-05
709 KSP preconditioned resid norm 7.225678145430e+00 true resid norm 1.153595518297e+04 ||r(i)||/||b|| 9.553586072851e-05
710 KSP preconditioned resid norm 7.225658847971e+00 true resid norm 1.153542952095e+04 ||r(i)||/||b|| 9.553150741986e-05
711 KSP preconditioned resid norm 7.225574123769e+00 true resid norm 1.153558436385e+04 ||r(i)||/||b|| 9.553278976268e-05
712 KSP preconditioned resid norm 7.225537396286e+00 true resid norm 1.153682591538e+04 ||r(i)||/||b|| 9.554307176292e-05
713 KSP preconditioned resid norm 7.225453520644e+00 true resid norm 1.153491010686e+04 ||r(i)||/||b|| 9.552720585391e-05
714 KSP preconditioned resid norm 7.225296158337e+00 true resid norm 1.153472510382e+04 ||r(i)||/||b|| 9.552567373759e-05
715 KSP preconditioned resid norm 7.225211046630e+00 true resid norm 1.153694288879e+04 ||r(i)||/||b|| 9.554404048681e-05
716 KSP preconditioned resid norm 7.224917290554e+00 true resid norm 1.153624306616e+04 ||r(i)||/||b|| 9.553824485428e-05
717 KSP preconditioned resid norm 7.224536823899e+00 true resid norm 1.153339847797e+04 ||r(i)||/||b|| 9.551468718811e-05
718 KSP preconditioned resid norm 7.224296204140e+00 true resid norm 1.153435702800e+04 ||r(i)||/||b|| 9.552262549068e-05
719 KSP preconditioned resid norm 7.224113583036e+00 true resid norm 1.153748406863e+04 ||r(i)||/||b|| 9.554852230743e-05
720 KSP preconditioned resid norm 7.224067785831e+00 true resid norm 1.153624136234e+04 ||r(i)||/||b|| 9.553823074399e-05
721 KSP preconditioned resid norm 7.224067774762e+00 true resid norm 1.153625793779e+04 ||r(i)||/||b|| 9.553836801478e-05
722 KSP preconditioned resid norm 7.224066897857e+00 true resid norm 1.153619993776e+04 ||r(i)||/||b|| 9.553788768327e-05
723 KSP preconditioned resid norm 7.224057882632e+00 true resid norm 1.153638674501e+04 ||r(i)||/||b|| 9.553943474126e-05
724 KSP preconditioned resid norm 7.224032657608e+00 true resid norm 1.153628595318e+04 ||r(i)||/||b|| 9.553860002627e-05
725 KSP preconditioned resid norm 7.223979309520e+00 true resid norm 1.153621116420e+04 ||r(i)||/||b|| 9.553798065590e-05
726 KSP preconditioned resid norm 7.223929515635e+00 true resid norm 1.153655688723e+04 ||r(i)||/||b|| 9.554084378654e-05
727 KSP preconditioned resid norm 7.223879217575e+00 true resid norm 1.153612769306e+04 ||r(i)||/||b|| 9.553728938347e-05
728 KSP preconditioned resid norm 7.223789581350e+00 true resid norm 1.153635494533e+04 ||r(i)||/||b|| 9.553917138990e-05
729 KSP preconditioned resid norm 7.223681683714e+00 true resid norm 1.153596830491e+04 ||r(i)||/||b|| 9.553596939877e-05
730 KSP preconditioned resid norm 7.223649024321e+00 true resid norm 1.153630343413e+04 ||r(i)||/||b|| 9.553874479605e-05
731 KSP preconditioned resid norm 7.223645905032e+00 true resid norm 1.153629509917e+04 ||r(i)||/||b|| 9.553867576950e-05
732 KSP preconditioned resid norm 7.223640965378e+00 true resid norm 1.153622575236e+04 ||r(i)||/||b|| 9.553810146879e-05
733 KSP preconditioned resid norm 7.223605269147e+00 true resid norm 1.153629232379e+04 ||r(i)||/||b|| 9.553865278497e-05
734 KSP preconditioned resid norm 7.223376065890e+00 true resid norm 1.153603199302e+04 ||r(i)||/||b|| 9.553649683655e-05
735 KSP preconditioned resid norm 7.222714289039e+00 true resid norm 1.153724257639e+04 ||r(i)||/||b|| 9.554652237173e-05
736 KSP preconditioned resid norm 7.222407770851e+00 true resid norm 1.153607656264e+04 ||r(i)||/||b|| 9.553686594314e-05
737 KSP preconditioned resid norm 7.222245287905e+00 true resid norm 1.153737688372e+04 ||r(i)||/||b|| 9.554763464771e-05
738 KSP preconditioned resid norm 7.222187700642e+00 true resid norm 1.153708008972e+04 ||r(i)||/||b|| 9.554517672646e-05
739 KSP preconditioned resid norm 7.222088925161e+00 true resid norm 1.153609504738e+04 ||r(i)||/||b|| 9.553701902591e-05
740 KSP preconditioned resid norm 7.221993239732e+00 true resid norm 1.153485802248e+04 ||r(i)||/||b|| 9.552677451324e-05
741 KSP preconditioned resid norm 7.221475687031e+00 true resid norm 1.153855736527e+04 ||r(i)||/||b|| 9.555741089251e-05
742 KSP preconditioned resid norm 7.221466988821e+00 true resid norm 1.153897084095e+04 ||r(i)||/||b|| 9.556083512171e-05
743 KSP preconditioned resid norm 7.220941538961e+00 true resid norm 1.153560931880e+04 ||r(i)||/||b|| 9.553299642895e-05
744 KSP preconditioned resid norm 7.220844958973e+00 true resid norm 1.153715476332e+04 ||r(i)||/||b|| 9.554579514131e-05
745 KSP preconditioned resid norm 7.220681137754e+00 true resid norm 1.153573144767e+04 ||r(i)||/||b|| 9.553400784815e-05
746 KSP preconditioned resid norm 7.220368017313e+00 true resid norm 1.153874004026e+04 ||r(i)||/||b|| 9.555892372880e-05
747 KSP preconditioned resid norm 7.220150983733e+00 true resid norm 1.153678033231e+04 ||r(i)||/||b|| 9.554269426340e-05
748 KSP preconditioned resid norm 7.219496403035e+00 true resid norm 1.154312910121e+04 ||r(i)||/||b|| 9.559527205967e-05
749 KSP preconditioned resid norm 7.218387179895e+00 true resid norm 1.153599405540e+04 ||r(i)||/||b|| 9.553618265342e-05
750 KSP preconditioned resid norm 7.217441555586e+00 true resid norm 1.154156063107e+04 ||r(i)||/||b|| 9.558228265897e-05
751 KSP preconditioned resid norm 7.217441295755e+00 true resid norm 1.154149925214e+04 ||r(i)||/||b|| 9.558177434478e-05
752 KSP preconditioned resid norm 7.217432084896e+00 true resid norm 1.154161589094e+04 ||r(i)||/||b|| 9.558274029761e-05
753 KSP preconditioned resid norm 7.217318605661e+00 true resid norm 1.154083404791e+04 ||r(i)||/||b|| 9.557626540708e-05
754 KSP preconditioned resid norm 7.217100980142e+00 true resid norm 1.154134326316e+04 ||r(i)||/||b|| 9.558048251062e-05
755 KSP preconditioned resid norm 7.216989453327e+00 true resid norm 1.154092350296e+04 ||r(i)||/||b|| 9.557700623565e-05
756 KSP preconditioned resid norm 7.216918504787e+00 true resid norm 1.154103227471e+04 ||r(i)||/||b|| 9.557790703693e-05
757 KSP preconditioned resid norm 7.216735297457e+00 true resid norm 1.154082849481e+04 ||r(i)||/||b|| 9.557621941872e-05
758 KSP preconditioned resid norm 7.216576950060e+00 true resid norm 1.154090196920e+04 ||r(i)||/||b|| 9.557682790227e-05
759 KSP preconditioned resid norm 7.216436014142e+00 true resid norm 1.154061335066e+04 ||r(i)||/||b|| 9.557443768658e-05
760 KSP preconditioned resid norm 7.216118685234e+00 true resid norm 1.154177001224e+04 ||r(i)||/||b|| 9.558401666447e-05
761 KSP preconditioned resid norm 7.215833494759e+00 true resid norm 1.154086630003e+04 ||r(i)||/||b|| 9.557653250538e-05
762 KSP preconditioned resid norm 7.215773914858e+00 true resid norm 1.154123106263e+04 ||r(i)||/||b|| 9.557955331371e-05
763 KSP preconditioned resid norm 7.215626119706e+00 true resid norm 1.154221400181e+04 ||r(i)||/||b|| 9.558769359673e-05
764 KSP preconditioned resid norm 7.215225341831e+00 true resid norm 1.154069925929e+04 ||r(i)||/||b|| 9.557514914526e-05
765 KSP preconditioned resid norm 7.214934785984e+00 true resid norm 1.154223553663e+04 ||r(i)||/||b|| 9.558787193895e-05
766 KSP preconditioned resid norm 7.214696647782e+00 true resid norm 1.154165808351e+04 ||r(i)||/||b|| 9.558308971848e-05
767 KSP preconditioned resid norm 7.214244644327e+00 true resid norm 1.154340639952e+04 ||r(i)||/||b|| 9.559756852604e-05
768 KSP preconditioned resid norm 7.213270504584e+00 true resid norm 1.154005229568e+04 ||r(i)||/||b|| 9.556979126857e-05
769 KSP preconditioned resid norm 7.211618669530e+00 true resid norm 1.154660918950e+04 ||r(i)||/||b|| 9.562409266665e-05
770 KSP preconditioned resid norm 7.210918439436e+00 true resid norm 1.154238696058e+04 ||r(i)||/||b|| 9.558912596749e-05
771 KSP preconditioned resid norm 7.210591893249e+00 true resid norm 1.154297499749e+04 ||r(i)||/||b|| 9.559399583842e-05
772 KSP preconditioned resid norm 7.210504987618e+00 true resid norm 1.154193445720e+04 ||r(i)||/||b|| 9.558537852753e-05
773 KSP preconditioned resid norm 7.210442506647e+00 true resid norm 1.154253156239e+04 ||r(i)||/||b|| 9.559032349800e-05
774 KSP preconditioned resid norm 7.210153266106e+00 true resid norm 1.154333808043e+04 ||r(i)||/||b|| 9.559700273646e-05
775 KSP preconditioned resid norm 7.210150308729e+00 true resid norm 1.154373826295e+04 ||r(i)||/||b|| 9.560031687738e-05
776 KSP preconditioned resid norm 7.209988796824e+00 true resid norm 1.154300100607e+04 ||r(i)||/||b|| 9.559421123036e-05
777 KSP preconditioned resid norm 7.209203000746e+00 true resid norm 1.153798294998e+04 ||r(i)||/||b|| 9.555265383007e-05
778 KSP preconditioned resid norm 7.208927099738e+00 true resid norm 1.153912496542e+04 ||r(i)||/||b|| 9.556211151479e-05
779 KSP preconditioned resid norm 7.208820457924e+00 true resid norm 1.154163776993e+04 ||r(i)||/||b|| 9.558292149008e-05
780 KSP preconditioned resid norm 7.208738383786e+00 true resid norm 1.153999134411e+04 ||r(i)||/||b|| 9.556928649362e-05
781 KSP preconditioned resid norm 7.208738361913e+00 true resid norm 1.154001355715e+04 ||r(i)||/||b|| 9.556947045258e-05
782 KSP preconditioned resid norm 7.208737818286e+00 true resid norm 1.153997294414e+04 ||r(i)||/||b|| 9.556913411292e-05
783 KSP preconditioned resid norm 7.208731152079e+00 true resid norm 1.154014258687e+04 ||r(i)||/||b|| 9.557053902163e-05
784 KSP preconditioned resid norm 7.208703090098e+00 true resid norm 1.154001566093e+04 ||r(i)||/||b|| 9.556948787518e-05
785 KSP preconditioned resid norm 7.208645884699e+00 true resid norm 1.153994412555e+04 ||r(i)||/||b|| 9.556889544967e-05
786 KSP preconditioned resid norm 7.208579379747e+00 true resid norm 1.154032621233e+04 ||r(i)||/||b|| 9.557205972938e-05
787 KSP preconditioned resid norm 7.208496506397e+00 true resid norm 1.153979131505e+04 ||r(i)||/||b|| 9.556762993825e-05
788 KSP preconditioned resid norm 7.208367305094e+00 true resid norm 1.154009451400e+04 ||r(i)||/||b|| 9.557014090268e-05
789 KSP preconditioned resid norm 7.208224000366e+00 true resid norm 1.153975086743e+04 ||r(i)||/||b|| 9.556729496836e-05
790 KSP preconditioned resid norm 7.208174189289e+00 true resid norm 1.153998393863e+04 ||r(i)||/||b|| 9.556922516458e-05
791 KSP preconditioned resid norm 7.208145263636e+00 true resid norm 1.154028067428e+04 ||r(i)||/||b|| 9.557168260271e-05
792 KSP preconditioned resid norm 7.208131496870e+00 true resid norm 1.153997090223e+04 ||r(i)||/||b|| 9.556911720268e-05
793 KSP preconditioned resid norm 7.208091238988e+00 true resid norm 1.153994483749e+04 ||r(i)||/||b|| 9.556890134564e-05
794 KSP preconditioned resid norm 7.207905606170e+00 true resid norm 1.153986196271e+04 ||r(i)||/||b|| 9.556821501209e-05
795 KSP preconditioned resid norm 7.207463235826e+00 true resid norm 1.153985841479e+04 ||r(i)||/||b|| 9.556818562966e-05
796 KSP preconditioned resid norm 7.207297827580e+00 true resid norm 1.154048889550e+04 ||r(i)||/||b|| 9.557340700201e-05
797 KSP preconditioned resid norm 7.207285047631e+00 true resid norm 1.154012281086e+04 ||r(i)||/||b|| 9.557037524521e-05
798 KSP preconditioned resid norm 7.207143006121e+00 true resid norm 1.154115678161e+04 ||r(i)||/||b|| 9.557893814995e-05
799 KSP preconditioned resid norm 7.206616871361e+00 true resid norm 1.154025876056e+04 ||r(i)||/||b|| 9.557150112265e-05
800 KSP preconditioned resid norm 7.206403657362e+00 true resid norm 1.153828666971e+04 ||r(i)||/||b|| 9.555516910729e-05
801 KSP preconditioned resid norm 7.206117021826e+00 true resid norm 1.154115981142e+04 ||r(i)||/||b|| 9.557896324153e-05
802 KSP preconditioned resid norm 7.206108040792e+00 true resid norm 1.154177366056e+04 ||r(i)||/||b|| 9.558404687827e-05
803 KSP preconditioned resid norm 7.205110428220e+00 true resid norm 1.153664984985e+04 ||r(i)||/||b|| 9.554161366331e-05
804 KSP preconditioned resid norm 7.204568369882e+00 true resid norm 1.154174811029e+04 ||r(i)||/||b|| 9.558383528189e-05
805 KSP preconditioned resid norm 7.204281983273e+00 true resid norm 1.154012594776e+04 ||r(i)||/||b|| 9.557040122362e-05
806 KSP preconditioned resid norm 7.203925117851e+00 true resid norm 1.154274070984e+04 ||r(i)||/||b|| 9.559205556799e-05
807 KSP preconditioned resid norm 7.203488692953e+00 true resid norm 1.153841841409e+04 ||r(i)||/||b|| 9.555626015808e-05
808 KSP preconditioned resid norm 7.203115676211e+00 true resid norm 1.154130399518e+04 ||r(i)||/||b|| 9.558015730996e-05
809 KSP preconditioned resid norm 7.202861676045e+00 true resid norm 1.153742344734e+04 ||r(i)||/||b|| 9.554802026779e-05
810 KSP preconditioned resid norm 7.202649582356e+00 true resid norm 1.153945343480e+04 ||r(i)||/||b|| 9.556483175816e-05
811 KSP preconditioned resid norm 7.202649502884e+00 true resid norm 1.153943082772e+04 ||r(i)||/||b|| 9.556464453597e-05
812 KSP preconditioned resid norm 7.202646421124e+00 true resid norm 1.153949967407e+04 ||r(i)||/||b|| 9.556521469206e-05
813 KSP preconditioned resid norm 7.202616415221e+00 true resid norm 1.153910529519e+04 ||r(i)||/||b|| 9.556194861437e-05
814 KSP preconditioned resid norm 7.202552594804e+00 true resid norm 1.153940983136e+04 ||r(i)||/||b|| 9.556447065301e-05
815 KSP preconditioned resid norm 7.202505594207e+00 true resid norm 1.153915036589e+04 ||r(i)||/||b|| 9.556232187066e-05
816 KSP preconditioned resid norm 7.202489548910e+00 true resid norm 1.153912974108e+04 ||r(i)||/||b|| 9.556215106485e-05
817 KSP preconditioned resid norm 7.202477557677e+00 true resid norm 1.153902088695e+04 ||r(i)||/||b|| 9.556124958136e-05
818 KSP preconditioned resid norm 7.202427700279e+00 true resid norm 1.153924738484e+04 ||r(i)||/||b|| 9.556312534027e-05
819 KSP preconditioned resid norm 7.202256815530e+00 true resid norm 1.153883568934e+04 ||r(i)||/||b|| 9.555971585372e-05
820 KSP preconditioned resid norm 7.201962506010e+00 true resid norm 1.153984412558e+04 ||r(i)||/||b|| 9.556806729258e-05
821 KSP preconditioned resid norm 7.201804961590e+00 true resid norm 1.153928739800e+04 ||r(i)||/||b|| 9.556345671215e-05
822 KSP preconditioned resid norm 7.201695229507e+00 true resid norm 1.153873693266e+04 ||r(i)||/||b|| 9.555889799297e-05
823 KSP preconditioned resid norm 7.201412697421e+00 true resid norm 1.154027837243e+04 ||r(i)||/||b|| 9.557166353972e-05
824 KSP preconditioned resid norm 7.201196616166e+00 true resid norm 1.153922075260e+04 ||r(i)||/||b|| 9.556290478338e-05
825 KSP preconditioned resid norm 7.201038639490e+00 true resid norm 1.154031849548e+04 ||r(i)||/||b|| 9.557199582175e-05
826 KSP preconditioned resid norm 7.200978120637e+00 true resid norm 1.153973979504e+04 ||r(i)||/||b|| 9.556720327151e-05
827 KSP preconditioned resid norm 7.200899200741e+00 true resid norm 1.154031025792e+04 ||r(i)||/||b|| 9.557192760179e-05
828 KSP preconditioned resid norm 7.200473040668e+00 true resid norm 1.153790830858e+04 ||r(i)||/||b|| 9.555203568182e-05
829 KSP preconditioned resid norm 7.199706526120e+00 true resid norm 1.154219332830e+04 ||r(i)||/||b|| 9.558752238751e-05
830 KSP preconditioned resid norm 7.199688522692e+00 true resid norm 1.154152585849e+04 ||r(i)||/||b|| 9.558199468726e-05
831 KSP preconditioned resid norm 7.199647267714e+00 true resid norm 1.154251028666e+04 ||r(i)||/||b|| 9.559014730149e-05
832 KSP preconditioned resid norm 7.199608263320e+00 true resid norm 1.154154565221e+04 ||r(i)||/||b|| 9.558215861042e-05
833 KSP preconditioned resid norm 7.199526978378e+00 true resid norm 1.153965353341e+04 ||r(i)||/||b|| 9.556648888949e-05
834 KSP preconditioned resid norm 7.199509725155e+00 true resid norm 1.154039406382e+04 ||r(i)||/||b|| 9.557262164652e-05
835 KSP preconditioned resid norm 7.199493908781e+00 true resid norm 1.154041578538e+04 ||r(i)||/||b|| 9.557280153524e-05
836 KSP preconditioned resid norm 7.199190097366e+00 true resid norm 1.154316225138e+04 ||r(i)||/||b|| 9.559554659526e-05
837 KSP preconditioned resid norm 7.198500829092e+00 true resid norm 1.154049845136e+04 ||r(i)||/||b|| 9.557348613963e-05
838 KSP preconditioned resid norm 7.198418895385e+00 true resid norm 1.154113268364e+04 ||r(i)||/||b|| 9.557873858081e-05
839 KSP preconditioned resid norm 7.198235438734e+00 true resid norm 1.154418367736e+04 ||r(i)||/||b|| 9.560400560958e-05
840 KSP preconditioned resid norm 7.197964207790e+00 true resid norm 1.154132722409e+04 ||r(i)||/||b|| 9.558034968188e-05
841 KSP preconditioned resid norm 7.197964124830e+00 true resid norm 1.154136063644e+04 ||r(i)||/||b|| 9.558062638867e-05
842 KSP preconditioned resid norm 7.197962521145e+00 true resid norm 1.154128979379e+04 ||r(i)||/||b|| 9.558003970008e-05
843 KSP preconditioned resid norm 7.197952136369e+00 true resid norm 1.154152599502e+04 ||r(i)||/||b|| 9.558199581794e-05
844 KSP preconditioned resid norm 7.197931086674e+00 true resid norm 1.154138915144e+04 ||r(i)||/||b|| 9.558086253774e-05
845 KSP preconditioned resid norm 7.197861353588e+00 true resid norm 1.154129023204e+04 ||r(i)||/||b|| 9.558004332945e-05
846 KSP preconditioned resid norm 7.197797146484e+00 true resid norm 1.154169541535e+04 ||r(i)||/||b|| 9.558339888487e-05
847 KSP preconditioned resid norm 7.197745589138e+00 true resid norm 1.154130042440e+04 ||r(i)||/||b|| 9.558012773829e-05
848 KSP preconditioned resid norm 7.197671256192e+00 true resid norm 1.154148005288e+04 ||r(i)||/||b|| 9.558161534469e-05
849 KSP preconditioned resid norm 7.197575733624e+00 true resid norm 1.154119483432e+04 ||r(i)||/||b|| 9.557925328623e-05
850 KSP preconditioned resid norm 7.197552367590e+00 true resid norm 1.154140499030e+04 ||r(i)||/||b|| 9.558099370849e-05
851 KSP preconditioned resid norm 7.197544095332e+00 true resid norm 1.154154155123e+04 ||r(i)||/||b|| 9.558212464783e-05
852 KSP preconditioned resid norm 7.197529387992e+00 true resid norm 1.154141914193e+04 ||r(i)||/||b|| 9.558111090622e-05
853 KSP preconditioned resid norm 7.197514362628e+00 true resid norm 1.154120030184e+04 ||r(i)||/||b|| 9.557929856595e-05
854 KSP preconditioned resid norm 7.197419246987e+00 true resid norm 1.154147811117e+04 ||r(i)||/||b|| 9.558159926431e-05
855 KSP preconditioned resid norm 7.197267339601e+00 true resid norm 1.154169369862e+04 ||r(i)||/||b|| 9.558338466759e-05
856 KSP preconditioned resid norm 7.197249338374e+00 true resid norm 1.154164954124e+04 ||r(i)||/||b|| 9.558301897502e-05
857 KSP preconditioned resid norm 7.197236169775e+00 true resid norm 1.154127164012e+04 ||r(i)||/||b|| 9.557988935915e-05
858 KSP preconditioned resid norm 7.197179492872e+00 true resid norm 1.154169694425e+04 ||r(i)||/||b|| 9.558341154657e-05
859 KSP preconditioned resid norm 7.197041372712e+00 true resid norm 1.154167559141e+04 ||r(i)||/||b|| 9.558323471146e-05
860 KSP preconditioned resid norm 7.196982481442e+00 true resid norm 1.154055830718e+04 ||r(i)||/||b|| 9.557398183993e-05
861 KSP preconditioned resid norm 7.196779105507e+00 true resid norm 1.154325291003e+04 ||r(i)||/||b|| 9.559629739152e-05
862 KSP preconditioned resid norm 7.196777399093e+00 true resid norm 1.154314356009e+04 ||r(i)||/||b|| 9.559539180194e-05
863 KSP preconditioned resid norm 7.196376986401e+00 true resid norm 1.153954183263e+04 ||r(i)||/||b|| 9.556556383124e-05
864 KSP preconditioned resid norm 7.196145709600e+00 true resid norm 1.154251131139e+04 ||r(i)||/||b|| 9.559015578790e-05
865 KSP preconditioned resid norm 7.196052567489e+00 true resid norm 1.154151854445e+04 ||r(i)||/||b|| 9.558193411549e-05
866 KSP preconditioned resid norm 7.195755970524e+00 true resid norm 1.154309590854e+04 ||r(i)||/||b|| 9.559499717212e-05
867 KSP preconditioned resid norm 7.195748011355e+00 true resid norm 1.154260375500e+04 ||r(i)||/||b|| 9.559092136640e-05
868 KSP preconditioned resid norm 7.195585550225e+00 true resid norm 1.154400139038e+04 ||r(i)||/||b|| 9.560249598653e-05
869 KSP preconditioned resid norm 7.195542633532e+00 true resid norm 1.154270413289e+04 ||r(i)||/||b|| 9.559175265333e-05
870 KSP preconditioned resid norm 7.195507038090e+00 true resid norm 1.154288316147e+04 ||r(i)||/||b|| 9.559323529163e-05
871 KSP preconditioned resid norm 7.195507025723e+00 true resid norm 1.154288609501e+04 ||r(i)||/||b|| 9.559325958598e-05
872 KSP preconditioned resid norm 7.195506578363e+00 true resid norm 1.154289470518e+04 ||r(i)||/||b|| 9.559333089171e-05
873 KSP preconditioned resid norm 7.195502575581e+00 true resid norm 1.154275749958e+04 ||r(i)||/||b|| 9.559219461344e-05
874 KSP preconditioned resid norm 7.195494110398e+00 true resid norm 1.154286596600e+04 ||r(i)||/||b|| 9.559309288608e-05
875 KSP preconditioned resid norm 7.195488172210e+00 true resid norm 1.154278710403e+04 ||r(i)||/||b|| 9.559243978492e-05
876 KSP preconditioned resid norm 7.195487182389e+00 true resid norm 1.154273738523e+04 ||r(i)||/||b|| 9.559202803498e-05
877 KSP preconditioned resid norm 7.195478713481e+00 true resid norm 1.154259407377e+04 ||r(i)||/||b|| 9.559084119060e-05
878 KSP preconditioned resid norm 7.195433627826e+00 true resid norm 1.154281835668e+04 ||r(i)||/||b|| 9.559269860603e-05
879 KSP preconditioned resid norm 7.195319916887e+00 true resid norm 1.154229450125e+04 ||r(i)||/||b|| 9.558836025880e-05
880 KSP preconditioned resid norm 7.195127322367e+00 true resid norm 1.154320321218e+04 ||r(i)||/||b|| 9.559588581510e-05
881 KSP preconditioned resid norm 7.194988399335e+00 true resid norm 1.154257062902e+04 ||r(i)||/||b|| 9.559064703118e-05
882 KSP preconditioned resid norm 7.194945666256e+00 true resid norm 1.154275921742e+04 ||r(i)||/||b|| 9.559220883992e-05
883 KSP preconditioned resid norm 7.194897298662e+00 true resid norm 1.154313246350e+04 ||r(i)||/||b|| 9.559529990469e-05
884 KSP preconditioned resid norm 7.194867135704e+00 true resid norm 1.154270594720e+04 ||r(i)||/||b|| 9.559176767866e-05
885 KSP preconditioned resid norm 7.194796615465e+00 true resid norm 1.154324278317e+04 ||r(i)||/||b|| 9.559621352517e-05
886 KSP preconditioned resid norm 7.194795457738e+00 true resid norm 1.154316106312e+04 ||r(i)||/||b|| 9.559553675456e-05
887 KSP preconditioned resid norm 7.194729511802e+00 true resid norm 1.154369865074e+04 ||r(i)||/||b|| 9.559998882600e-05
888 KSP preconditioned resid norm 7.194540520451e+00 true resid norm 1.154207998367e+04 ||r(i)||/||b|| 9.558658371564e-05
889 KSP preconditioned resid norm 7.194298564258e+00 true resid norm 1.154457402609e+04 ||r(i)||/||b|| 9.560723831124e-05
890 KSP preconditioned resid norm 7.194211215388e+00 true resid norm 1.154315634506e+04 ||r(i)||/||b|| 9.559549768161e-05
891 KSP preconditioned resid norm 7.194087112452e+00 true resid norm 1.154485424586e+04 ||r(i)||/||b|| 9.560955897189e-05
892 KSP preconditioned resid norm 7.194055182294e+00 true resid norm 1.154365904537e+04 ||r(i)||/||b|| 9.559966083115e-05
893 KSP preconditioned resid norm 7.194048944021e+00 true resid norm 1.154418649709e+04 ||r(i)||/||b|| 9.560402896138e-05
894 KSP preconditioned resid norm 7.193939682211e+00 true resid norm 1.154393638054e+04 ||r(i)||/||b|| 9.560195760278e-05
895 KSP preconditioned resid norm 7.193904216251e+00 true resid norm 1.154394875483e+04 ||r(i)||/||b|| 9.560206008134e-05
896 KSP preconditioned resid norm 7.193849335787e+00 true resid norm 1.154517596336e+04 ||r(i)||/||b|| 9.561222329904e-05
897 KSP preconditioned resid norm 7.193568034171e+00 true resid norm 1.154207786340e+04 ||r(i)||/||b|| 9.558656615652e-05
898 KSP preconditioned resid norm 7.193282272126e+00 true resid norm 1.154408917503e+04 ||r(i)||/||b|| 9.560322298157e-05
899 KSP preconditioned resid norm 7.193156669079e+00 true resid norm 1.154638340356e+04 ||r(i)||/||b|| 9.562222280375e-05
900 KSP preconditioned resid norm 7.193075068262e+00 true resid norm 1.154495596218e+04 ||r(i)||/||b|| 9.561040134303e-05
901 KSP preconditioned resid norm 7.193075042941e+00 true resid norm 1.154497417770e+04 ||r(i)||/||b|| 9.561055219625e-05
902 KSP preconditioned resid norm 7.193074171495e+00 true resid norm 1.154491924051e+04 ||r(i)||/||b|| 9.561009722984e-05
903 KSP preconditioned resid norm 7.193067789121e+00 true resid norm 1.154507875685e+04 ||r(i)||/||b|| 9.561141827613e-05
904 KSP preconditioned resid norm 7.193050778483e+00 true resid norm 1.154498708177e+04 ||r(i)||/||b|| 9.561065906224e-05
905 KSP preconditioned resid norm 7.193002778314e+00 true resid norm 1.154485825675e+04 ||r(i)||/||b|| 9.560959218837e-05
906 KSP preconditioned resid norm 7.192960235451e+00 true resid norm 1.154519139248e+04 ||r(i)||/||b|| 9.561235107640e-05
907 KSP preconditioned resid norm 7.192911779696e+00 true resid norm 1.154483390122e+04 ||r(i)||/||b|| 9.560939048626e-05
908 KSP preconditioned resid norm 7.192843974924e+00 true resid norm 1.154496164487e+04 ||r(i)||/||b|| 9.561044840466e-05
909 KSP preconditioned resid norm 7.192754925480e+00 true resid norm 1.154471634806e+04 ||r(i)||/||b|| 9.560841696113e-05
910 KSP preconditioned resid norm 7.192718938357e+00 true resid norm 1.154498297292e+04 ||r(i)||/||b|| 9.561062503453e-05
911 KSP preconditioned resid norm 7.192717355649e+00 true resid norm 1.154505965933e+04 ||r(i)||/||b|| 9.561126011863e-05
912 KSP preconditioned resid norm 7.192714425342e+00 true resid norm 1.154503861039e+04 ||r(i)||/||b|| 9.561108580033e-05
913 KSP preconditioned resid norm 7.192689064287e+00 true resid norm 1.154479936883e+04 ||r(i)||/||b|| 9.560910450376e-05
914 KSP preconditioned resid norm 7.192599141460e+00 true resid norm 1.154493241164e+04 ||r(i)||/||b|| 9.561020630751e-05
915 KSP preconditioned resid norm 7.192411197035e+00 true resid norm 1.154564281955e+04 ||r(i)||/||b|| 9.561608960287e-05
916 KSP preconditioned resid norm 7.192352667169e+00 true resid norm 1.154528433915e+04 ||r(i)||/||b|| 9.561312082106e-05
917 KSP preconditioned resid norm 7.192329563538e+00 true resid norm 1.154494212972e+04 ||r(i)||/||b|| 9.561028678853e-05
918 KSP preconditioned resid norm 7.192253245672e+00 true resid norm 1.154521586285e+04 ||r(i)||/||b|| 9.561255372961e-05
919 KSP preconditioned resid norm 7.192072505396e+00 true resid norm 1.154572973092e+04 ||r(i)||/||b|| 9.561680936572e-05
920 KSP preconditioned resid norm 7.191931943165e+00 true resid norm 1.154397394083e+04 ||r(i)||/||b|| 9.560226866108e-05
921 KSP preconditioned resid norm 7.191564889267e+00 true resid norm 1.154744137817e+04 ||r(i)||/||b|| 9.563098449826e-05
922 KSP preconditioned resid norm 7.191549910577e+00 true resid norm 1.154730486037e+04 ||r(i)||/||b|| 9.562985391611e-05
923 KSP preconditioned resid norm 7.191317840963e+00 true resid norm 1.154421491551e+04 ||r(i)||/||b|| 9.560426431060e-05
924 KSP preconditioned resid norm 7.191112349985e+00 true resid norm 1.154625128761e+04 ||r(i)||/||b|| 9.562112867585e-05
925 KSP preconditioned resid norm 7.191082327161e+00 true resid norm 1.154491053240e+04 ||r(i)||/||b|| 9.561002511300e-05
926 KSP preconditioned resid norm 7.190789668045e+00 true resid norm 1.154779805734e+04 ||r(i)||/||b|| 9.563393836306e-05
927 KSP preconditioned resid norm 7.190405784691e+00 true resid norm 1.154376635260e+04 ||r(i)||/||b|| 9.560054950390e-05
928 KSP preconditioned resid norm 7.189129222975e+00 true resid norm 1.155355870137e+04 ||r(i)||/||b|| 9.568164555993e-05
929 KSP preconditioned resid norm 7.188405933667e+00 true resid norm 1.154789471298e+04 ||r(i)||/||b|| 9.563473882380e-05
930 KSP preconditioned resid norm 7.188037370163e+00 true resid norm 1.155089435911e+04 ||r(i)||/||b|| 9.565958061369e-05
931 KSP preconditioned resid norm 7.188037249332e+00 true resid norm 1.155085543429e+04 ||r(i)||/||b|| 9.565925825498e-05
932 KSP preconditioned resid norm 7.188031791565e+00 true resid norm 1.155093318228e+04 ||r(i)||/||b|| 9.565990213067e-05
933 KSP preconditioned resid norm 7.187957874026e+00 true resid norm 1.155036342092e+04 ||r(i)||/||b|| 9.565518361007e-05
934 KSP preconditioned resid norm 7.187796258843e+00 true resid norm 1.155079699935e+04 ||r(i)||/||b|| 9.565877432170e-05
935 KSP preconditioned resid norm 7.187667068173e+00 true resid norm 1.155028768895e+04 ||r(i)||/||b|| 9.565455643021e-05
936 KSP preconditioned resid norm 7.187577783238e+00 true resid norm 1.155048664693e+04 ||r(i)||/||b|| 9.565620411529e-05
937 KSP preconditioned resid norm 7.187368041620e+00 true resid norm 1.155038877933e+04 ||r(i)||/||b|| 9.565539361761e-05
938 KSP preconditioned resid norm 7.187127474179e+00 true resid norm 1.155041142808e+04 ||r(i)||/||b|| 9.565558118488e-05
939 KSP preconditioned resid norm 7.186976828036e+00 true resid norm 1.155032947593e+04 ||r(i)||/||b|| 9.565490249210e-05
940 KSP preconditioned resid norm 7.186867893396e+00 true resid norm 1.155094660671e+04 ||r(i)||/||b|| 9.566001330607e-05
941 KSP preconditioned resid norm 7.186795028794e+00 true resid norm 1.155046157477e+04 ||r(i)||/||b|| 9.565599647839e-05
942 KSP preconditioned resid norm 7.186786183940e+00 true resid norm 1.155063228000e+04 ||r(i)||/||b|| 9.565741018632e-05
943 KSP preconditioned resid norm 7.186600734712e+00 true resid norm 1.155188485310e+04 ||r(i)||/||b|| 9.566778346249e-05
944 KSP preconditioned resid norm 7.185914635398e+00 true resid norm 1.154983615930e+04 ||r(i)||/||b|| 9.565081705425e-05
945 KSP preconditioned resid norm 7.184966283348e+00 true resid norm 1.155274663883e+04 ||r(i)||/||b|| 9.567492040439e-05
946 KSP preconditioned resid norm 7.184850921828e+00 true resid norm 1.155155604594e+04 ||r(i)||/||b|| 9.566506042183e-05
947 KSP preconditioned resid norm 7.184693468580e+00 true resid norm 1.155266445243e+04 ||r(i)||/||b|| 9.567423977164e-05
948 KSP preconditioned resid norm 7.184431937699e+00 true resid norm 1.155067194497e+04 ||r(i)||/||b|| 9.565773867470e-05
949 KSP preconditioned resid norm 7.183714853342e+00 true resid norm 1.155488372460e+04 ||r(i)||/||b|| 9.569261883721e-05
950 KSP preconditioned resid norm 7.183229824086e+00 true resid norm 1.155141559527e+04 ||r(i)||/||b|| 9.566389726925e-05
951 KSP preconditioned resid norm 7.183093300729e+00 true resid norm 1.155241648075e+04 ||r(i)||/||b|| 9.567218617597e-05
952 KSP preconditioned resid norm 7.183061323905e+00 true resid norm 1.155294836589e+04 ||r(i)||/||b|| 9.567659102180e-05
953 KSP preconditioned resid norm 7.182864969030e+00 true resid norm 1.155098411601e+04 ||r(i)||/||b|| 9.566032394211e-05
954 KSP preconditioned resid norm 7.182782821182e+00 true resid norm 1.155297900664e+04 ||r(i)||/||b|| 9.567684477545e-05
955 KSP preconditioned resid norm 7.182655649388e+00 true resid norm 1.155384388809e+04 ||r(i)||/||b|| 9.568400735477e-05
956 KSP preconditioned resid norm 7.182295799609e+00 true resid norm 1.155256304358e+04 ||r(i)||/||b|| 9.567339994680e-05
957 KSP preconditioned resid norm 7.181399956169e+00 true resid norm 1.154821898666e+04 ||r(i)||/||b|| 9.563742432011e-05
958 KSP preconditioned resid norm 7.180785504313e+00 true resid norm 1.154994838740e+04 ||r(i)||/||b|| 9.565174647949e-05
959 KSP preconditioned resid norm 7.180751030489e+00 true resid norm 1.155127286764e+04 ||r(i)||/||b|| 9.566271525990e-05
960 KSP preconditioned resid norm 7.180501811163e+00 true resid norm 1.155074516689e+04 ||r(i)||/||b|| 9.565834506742e-05
961 KSP preconditioned resid norm 7.180501755409e+00 true resid norm 1.155074194046e+04 ||r(i)||/||b|| 9.565831834743e-05
962 KSP preconditioned resid norm 7.180501216344e+00 true resid norm 1.155071487943e+04 ||r(i)||/||b|| 9.565809423955e-05
963 KSP preconditioned resid norm 7.180496967397e+00 true resid norm 1.155085679516e+04 ||r(i)||/||b|| 9.565926952508e-05
964 KSP preconditioned resid norm 7.180489739989e+00 true resid norm 1.155077391091e+04 ||r(i)||/||b|| 9.565858311309e-05
965 KSP preconditioned resid norm 7.180441735075e+00 true resid norm 1.155069206506e+04 ||r(i)||/||b|| 9.565790530067e-05
966 KSP preconditioned resid norm 7.180384447291e+00 true resid norm 1.155101338999e+04 ||r(i)||/||b|| 9.566056637668e-05
967 KSP preconditioned resid norm 7.180351358353e+00 true resid norm 1.155066924091e+04 ||r(i)||/||b|| 9.565771628082e-05
968 KSP preconditioned resid norm 7.180270183541e+00 true resid norm 1.155090235506e+04 ||r(i)||/||b|| 9.565964683275e-05
969 KSP preconditioned resid norm 7.180115994184e+00 true resid norm 1.155019810268e+04 ||r(i)||/||b|| 9.565381451491e-05
970 KSP preconditioned resid norm 7.180009154338e+00 true resid norm 1.155084062398e+04 ||r(i)||/||b|| 9.565913560228e-05
971 KSP preconditioned resid norm 7.179958528992e+00 true resid norm 1.155054435115e+04 ||r(i)||/||b|| 9.565668199710e-05
972 KSP preconditioned resid norm 7.179823386858e+00 true resid norm 1.155108073886e+04 ||r(i)||/||b|| 9.566112413127e-05
973 KSP preconditioned resid norm 7.179765969486e+00 true resid norm 1.155045672675e+04 ||r(i)||/||b|| 9.565595632912e-05
974 KSP preconditioned resid norm 7.179731391564e+00 true resid norm 1.155070670352e+04 ||r(i)||/||b|| 9.565802653015e-05
975 KSP preconditioned resid norm 7.179586623691e+00 true resid norm 1.155151982917e+04 ||r(i)||/||b|| 9.566476049000e-05
976 KSP preconditioned resid norm 7.179494877732e+00 true resid norm 1.155064282515e+04 ||r(i)||/||b|| 9.565749751676e-05
977 KSP preconditioned resid norm 7.179494504063e+00 true resid norm 1.155066416907e+04 ||r(i)||/||b|| 9.565767427799e-05
978 KSP preconditioned resid norm 7.179299259051e+00 true resid norm 1.155147393983e+04 ||r(i)||/||b|| 9.566438045408e-05
979 KSP preconditioned resid norm 7.178869548476e+00 true resid norm 1.155190457319e+04 ||r(i)||/||b|| 9.566794677587e-05
980 KSP preconditioned resid norm 7.178438452513e+00 true resid norm 1.154872746538e+04 ||r(i)||/||b|| 9.564163532403e-05
981 KSP preconditioned resid norm 7.177807609599e+00 true resid norm 1.155346489225e+04 ||r(i)||/||b|| 9.568086867282e-05
982 KSP preconditioned resid norm 7.177570985262e+00 true resid norm 1.155091038091e+04 ||r(i)||/||b|| 9.565971329942e-05
983 KSP preconditioned resid norm 7.177489302414e+00 true resid norm 1.155006729504e+04 ||r(i)||/||b|| 9.565273122182e-05
984 KSP preconditioned resid norm 7.177319725031e+00 true resid norm 1.155067379856e+04 ||r(i)||/||b|| 9.565775402528e-05
985 KSP preconditioned resid norm 7.177262324113e+00 true resid norm 1.155260023983e+04 ||r(i)||/||b|| 9.567370799026e-05
986 KSP preconditioned resid norm 7.177213457621e+00 true resid norm 1.155390688279e+04 ||r(i)||/||b|| 9.568452904997e-05
987 KSP preconditioned resid norm 7.176807016638e+00 true resid norm 1.154943164188e+04 ||r(i)||/||b|| 9.564746701350e-05
988 KSP preconditioned resid norm 7.176373599022e+00 true resid norm 1.155381826289e+04 ||r(i)||/||b|| 9.568379513779e-05
989 KSP preconditioned resid norm 7.176161377528e+00 true resid norm 1.155105823808e+04 ||r(i)||/||b|| 9.566093778945e-05
990 KSP preconditioned resid norm 7.176052334447e+00 true resid norm 1.155172990229e+04 ||r(i)||/||b|| 9.566650022597e-05
991 KSP preconditioned resid norm 7.176052309384e+00 true resid norm 1.155172173347e+04 ||r(i)||/||b|| 9.566643257531e-05
992 KSP preconditioned resid norm 7.176050576185e+00 true resid norm 1.155176352631e+04 ||r(i)||/||b|| 9.566677868579e-05
993 KSP preconditioned resid norm 7.176029537659e+00 true resid norm 1.155146593409e+04 ||r(i)||/||b|| 9.566431415395e-05
994 KSP preconditioned resid norm 7.175975423474e+00 true resid norm 1.155173237968e+04 ||r(i)||/||b|| 9.566652074264e-05
995 KSP preconditioned resid norm 7.175878317194e+00 true resid norm 1.155128118357e+04 ||r(i)||/||b|| 9.566278412893e-05
996 KSP preconditioned resid norm 7.175769491348e+00 true resid norm 1.155138284381e+04 ||r(i)||/||b|| 9.566362603565e-05
997 KSP preconditioned resid norm 7.175668551495e+00 true resid norm 1.155147770038e+04 ||r(i)||/||b|| 9.566441159729e-05
998 KSP preconditioned resid norm 7.175520926979e+00 true resid norm 1.155139383459e+04 ||r(i)||/||b|| 9.566371705660e-05
999 KSP preconditioned resid norm 7.175332227452e+00 true resid norm 1.155161520017e+04 ||r(i)||/||b|| 9.566555031191e-05
1000 KSP preconditioned resid norm 7.175242959255e+00 true resid norm 1.155179363599e+04 ||r(i)||/||b|| 9.566702804132e-05
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=1000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.   0.   0.  
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
          Symmetric graph false
          Number of levels to square graph 1
          Number smoothing steps 1
        Complexity:    grid = 1.31821
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 1 MPI processes
      type: bjacobi
        number of blocks = 1
        Local solve is same for all blocks, in the following KSP and PC objects:
        KSP Object: (mg_coarse_sub_) 1 MPI processes
          type: preonly
          maximum iterations=1, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
          left preconditioning
          using NONE norm type for convergence test
        PC Object: (mg_coarse_sub_) 1 MPI processes
          type: lu
            out-of-place factorization
            tolerance for zero pivot 2.22045e-14
            using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
            matrix ordering: nd
            factor fill ratio given 5., needed 1.
              Factored matrix follows:
                Mat Object: 1 MPI processes
                  type: seqaij
                  rows=17, cols=17
                  package used to perform factorization: petsc
                  total: nonzeros=163, allocated nonzeros=163
                  total number of mallocs used during MatSetValues calls =0
                    using I-node routines: found 8 nodes, limit used is 5
          linear system matrix = precond matrix:
          Mat Object: 1 MPI processes
            type: seqaij
            rows=17, cols=17
            total: nonzeros=163, allocated nonzeros=163
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 12 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=17, cols=17
        total: nonzeros=163, allocated nonzeros=163
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 12 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 1 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.100647, max = 1.10712
        eigenvalues estimate via gmres min 0.750571, max 1.00647
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_1_esteig_) 1 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=100, cols=100
        total: nonzeros=1240, allocated nonzeros=1240
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 1 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.11638, max = 1.28018
        eigenvalues estimate via gmres min 0.127872, max 1.1638
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_2_esteig_) 1 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=537, cols=537
        total: nonzeros=5291, allocated nonzeros=5291
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 1 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.1, max = 1.1
        eigenvalues estimate via gmres min 0.0705685, max 1.
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_3_esteig_) 1 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=1541, cols=1541
        total: nonzeros=8039, allocated nonzeros=8039
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 1 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.0998151, max = 1.09797
        eigenvalues estimate via gmres min 0.0132089, max 0.998151
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_4_esteig_) 1 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=8541, cols=8541
        total: nonzeros=46299, allocated nonzeros=46299
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 5464 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
    type: seqaij
    rows=8541, cols=8541
    total: nonzeros=46299, allocated nonzeros=46299
    total number of mallocs used during MatSetValues calls =0
      using I-node routines: found 5464 nodes, limit used is 5
-------------- next part --------------
A non-text attachment was scrubbed...
Name: solveYmat.zip
Type: application/zip
Size: 344371 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/05296caa/attachment-0001.zip>

From mfadams at lbl.gov  Fri Feb  1 08:12:52 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 1 Feb 2019 09:12:52 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
Message-ID: <CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>

>
> Both GAMG and ILU are nice and dandy for this,
>

I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
converge it way down, say rtol = 1.e-12. See which one is better in the
early iteration and pick it. It would be nice to check that it solves the
problem ...

The residual drops about 5 orders in the first iteration and then
flatlines. That is very bad. Check that the smoother can actually solve the
problem.


> but as soon as I look at a bigger system, like a network with 8500 buses,
> the out-of-box gamg craps out. I am not sure where to start when it comes
> to tuning GAMG.
>

First try -pc_gamg_nsmooths 0

You can run with -info and grep on GAMG to see some diagnostic output.
Eigen estimates are fragile in practice and with this parameter and SOR
there are no eigen estimates needed. The max eigen values for all levels
should be between say 2 (or less) and say 3-4. Much higher is a sign of a
problem.


> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
>
> I'm also attaching a zip file which contains the simple PETSc script that
> loads the binary matrix/vector as well as two test cases, if you guys are
> interested in trying it out. It only works if you have PETSc configured
> with complex numbers.
>
> Thanks
>
> Justin
>
> PS - A couple years ago I had asked if there was a paper/tutorial on
> using/tuning GAMG. Does such a thing exist today?
>

There is a write up in the manual that is tutorial like.


>
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>> solve:
>>>
>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>
>>> Right now we're just trying to solve eq(5) (in section 1), inverting the
>>> linear Y-bus matrix. Eventually we have to be able to solve equations like
>>> those in the next section.
>>>
>>
>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>> M-matrix to me (if all the y's are positive). This means
>> that it should be really easy to solve, and I think GAMG should do it.
>> You can start out just doing relaxation, like SOR, on
>> small examples.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm working with some folks to extract a linear system of equations
>>>>> from an external software package that solves power flow equations in
>>>>> complex form. Since that external package uses serial direct solvers like
>>>>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>>>>> be solved in PETSc using its parallel solvers.
>>>>>
>>>>> I got mumps to achieve a very minor speedup across two MPI processes
>>>>> on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>>>>> seconds). However I want to use iterative solvers and preconditioners but I
>>>>> have never worked with complex numbers so I am not sure what the "best"
>>>>> options are given PETSc's capabilities.
>>>>>
>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>> claim to successfully precondition power flow equations operate in this
>>>>> form.
>>>>>
>>>>
>>>> 1) We really need to see the (simplified) equations
>>>>
>>>> 2) All complex equations can be converted to a system of real equations
>>>> twice as large, but this is not necessarily the best way to go
>>>>
>>>>  Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Justin
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/0febddff/attachment.html>

From jychang48 at gmail.com  Fri Feb  1 14:04:45 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 1 Feb 2019 13:04:45 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
Message-ID: <CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>

Hi Mark,

1) So with these options:

-ksp_type gmres
-ksp_rtol 1e-15
-ksp_monitor_true_residual
-ksp_converged_reason
-pc_type bjacobi

This is what I get:

  0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
  2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
  3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
  4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
  5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
Linear solve converged due to CONVERGED_RTOL iterations 5

2) With richardson/sor:

-ksp_type richardson
-ksp_rtol 1e-15
-ksp_monitor_true_residual
-pc_type sor

This is what I get:

  0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
  2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
  3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
  4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
  5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
  6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
  7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
  8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
  9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
 10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
 11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
 12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
 13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
 14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
 15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
 16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
 17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
 18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
 19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
 20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
 21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
 22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
 23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
 24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
 25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
 26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
 27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
 28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
 29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
 30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
 31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
 32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
 33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
 34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
 35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
 36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
 37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
 38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
 39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
 40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
Linear solve converged due to CONVERGED_RTOL iterations 40

3) And lastly with chebyshev/jacobi:

-ksp_type chebyshev
-ksp_rtol 1e-15
-ksp_monitor_true_residual
-pc_type jacobi

  0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
  1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
  2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
  3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
  4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
  5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
  6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
  7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
  8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
  9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
 10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
 11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
 12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
 13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
 14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
 15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
 16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
 17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
 18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
 19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
 20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
 21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
 22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
 23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
 24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
 25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
 26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
 27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
 28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
 29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
 30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
 31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
 32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
 33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
 34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
 35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
 36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
 37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
 38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
 39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
 40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
 41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
 42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
 43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
 44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
 45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
 46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
 47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
 48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
 49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
 50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
 51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
 52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
 53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
 54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
 55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
 56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
 57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
 58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
 59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
 60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
 61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
 62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
 63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
 64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
 65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
 66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
Linear solve converged due to CONVERGED_RTOL iterations 67

Looks like neither of these ksp/pc combos are good? I also tried
-pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
GAMG info I was able to grep from the out-of-box params:

[0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row
(ave)=5, np=1
[0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold 0.,
5.42079 nnz ave. (N=8541)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 1541 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
min=2.394056e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
5.21674 nnz ave. (N=1541)
[0] PCGAMGProlongator_AGG(): New grid 537 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
min=6.783380e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
9.85289 nnz ave. (N=537)
[0] PCGAMGProlongator_AGG(): New grid 100 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
min=5.974776e-02 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0., 12.4
nnz ave. (N=100)
[0] PCGAMGProlongator_AGG(): New grid 17 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
min=4.842076e-01 PC=jacobi
[0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821

And the one with pc_gamg_agg_nsmooths 0

[0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row
(ave)=5, np=1
[0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold 0.,
5.42079 nnz ave. (N=8541)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 1541 nodes
[0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
[0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold 0.,
3.07398 nnz ave. (N=1541)
[0] PCGAMGProlongator_AGG(): New grid 814 nodes
[0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
3.02211 nnz ave. (N=814)
[0] PCGAMGProlongator_AGG(): New grid 461 nodes
[0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
3.00434 nnz ave. (N=461)
[0] PCGAMGProlongator_AGG(): New grid 290 nodes
[0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0., 3.
nnz ave. (N=290)
[0] PCGAMGProlongator_AGG(): New grid 197 nodes
[0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0., 3.
nnz ave. (N=197)
[0] PCGAMGProlongator_AGG(): New grid 127 nodes
[0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
2.98425 nnz ave. (N=127)
[0] PCGAMGProlongator_AGG(): New grid 82 nodes
[0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
2.97561 nnz ave. (N=82)
[0] PCGAMGProlongator_AGG(): New grid 66 nodes
[0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
[0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
2.9697 nnz ave. (N=66)
[0] PCGAMGProlongator_AGG(): New grid 36 nodes
[0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
[0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689


Thanks,
Justin

On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:

>
>
>> Both GAMG and ILU are nice and dandy for this,
>>
>
> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
> converge it way down, say rtol = 1.e-12. See which one is better in the
> early iteration and pick it. It would be nice to check that it solves the
> problem ...
>
> The residual drops about 5 orders in the first iteration and then
> flatlines. That is very bad. Check that the smoother can actually solve the
> problem.
>
>
>> but as soon as I look at a bigger system, like a network with 8500 buses,
>> the out-of-box gamg craps out. I am not sure where to start when it comes
>> to tuning GAMG.
>>
>
> First try -pc_gamg_nsmooths 0
>
> You can run with -info and grep on GAMG to see some diagnostic output.
> Eigen estimates are fragile in practice and with this parameter and SOR
> there are no eigen estimates needed. The max eigen values for all levels
> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
> problem.
>
>
>> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
>>
>> I'm also attaching a zip file which contains the simple PETSc script that
>> loads the binary matrix/vector as well as two test cases, if you guys are
>> interested in trying it out. It only works if you have PETSc configured
>> with complex numbers.
>>
>> Thanks
>>
>> Justin
>>
>> PS - A couple years ago I had asked if there was a paper/tutorial on
>> using/tuning GAMG. Does such a thing exist today?
>>
>
> There is a write up in the manual that is tutorial like.
>
>
>>
>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>>> solve:
>>>>
>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>
>>>> Right now we're just trying to solve eq(5) (in section 1), inverting
>>>> the linear Y-bus matrix. Eventually we have to be able to solve equations
>>>> like those in the next section.
>>>>
>>>
>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>> M-matrix to me (if all the y's are positive). This means
>>> that it should be really easy to solve, and I think GAMG should do it.
>>> You can start out just doing relaxation, like SOR, on
>>> small examples.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm working with some folks to extract a linear system of equations
>>>>>> from an external software package that solves power flow equations in
>>>>>> complex form. Since that external package uses serial direct solvers like
>>>>>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>>>>>> be solved in PETSc using its parallel solvers.
>>>>>>
>>>>>> I got mumps to achieve a very minor speedup across two MPI processes
>>>>>> on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>>>>>> seconds). However I want to use iterative solvers and preconditioners but I
>>>>>> have never worked with complex numbers so I am not sure what the "best"
>>>>>> options are given PETSc's capabilities.
>>>>>>
>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>> form.
>>>>>>
>>>>>
>>>>> 1) We really need to see the (simplified) equations
>>>>>
>>>>> 2) All complex equations can be converted to a system of real
>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>
>>>>>  Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> Justin
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/73951277/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Fri Feb  1 14:20:40 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 1 Feb 2019 14:20:40 -0600
Subject: [petsc-users] Reading a complex vector from HDF5
Message-ID: <CAOGsD9hZfbk28bKP+hi0bfXuJUiCSh3k1P3a1QJ4FyOP8pACrA@mail.gmail.com>

Hi,

I'm trying to load a complex vector from hdf5 and I get and I get the
following error:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: File contains real numbers but PETSc is configured for
complex. The conversion is not yet implemented. Configure with
--with-scalar-type=real.
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision:
ba892ce6f06cbab637e31ab2bf33fd996c92d19d  GIT Date: 2019-01-29 05:55:16
+0100
[0]PETSC ERROR: ./ex_modify on a  named xrm by sajid Fri Feb  1 14:18:30
2019
[0]PETSC ERROR: Configure options
--prefix=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/petsc-develop-gh5lollh4dzklppm7kb7wpjtyh4gi4mi
--with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0
CFLAGS="-march=native -O2" FFLAGS="-march=native -O2"
CXXFLAGS="-march=native -O2"
--with-cc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpicc
--with-cxx=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpic++
--with-fc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpif90
--with-precision=double --with-scalar-type=complex
--with-shared-libraries=1 --with-debugging=0 --with-64-bit-indices=0
COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS=
--with-blaslapack-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/openblas-0.3.5-heqggft2eii2xktdbyp2zovez2yj6ugd/lib/libopenblas.so
--with-x=1 --with-clanguage=C --with-scalapack=0 --with-metis=1
--with-metis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/metis-5.1.0-mugo6fn7u2nlz72vo4ajlhbba5q7a5jn
--with-hdf5=1
--with-hdf5-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/hdf5-1.10.4-pes54zx5fsbgdg2nr7reyopc4zfc43wo
--with-hypre=0 --with-parmetis=1
--with-parmetis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/parmetis-4.0.3-kizvrkoos4ddlfc7jsnatvh3mdk3lgvg
--with-mumps=0 --with-trilinos=0 --with-cxx-dialect=C++11
--with-superlu_dist-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/superlu-dist-develop-d6dqq3ribpkyoawnwuckbtydgddijj6j/include
--with-superlu_dist-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/superlu-dist-develop-d6dqq3ribpkyoawnwuckbtydgddijj6j/lib/libsuperlu_dist.a
--with-superlu_dist=1 --with-suitesparse=0
--with-zlib-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/zlib-1.2.11-v4lhujkxdy6ukr5ymjxgxcfb2qoo6vf3/include
--with-zlib-lib="-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/zlib-1.2.11-v4lhujkxdy6ukr5ymjxgxcfb2qoo6vf3/lib
-lz" --with-zlib=1
[0]PETSC ERROR: #1 PetscViewerHDF5Load() line 1348 in
/tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c
[0]PETSC ERROR: #2 VecLoad_HDF5() line 170 in
/tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #3 VecLoad_Default() line 288 in
/tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #4 VecLoad() line 933 in
/tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #5 main() line 132 in
/raid/home/sajid/packages/xwp_petsc/2d/matter_repeat/ex_modify.c
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------

As per src/vec/vec/examples/tutorials/ex10.c , a complex 1D vector is saves
as a 2D real valued vector of shape dim,2. I've done the same with my data
and yet I get the above error. Is there any special precaution I should
take when making the HDF5 file ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/ce8f8eba/attachment.html>

From knepley at gmail.com  Fri Feb  1 14:21:53 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 1 Feb 2019 15:21:53 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
Message-ID: <CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>

On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:

> Hi Mark,
>
> 1) So with these options:
>
> -ksp_type gmres
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -ksp_converged_reason
> -pc_type bjacobi
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> Linear solve converged due to CONVERGED_RTOL iterations 5
>
> 2) With richardson/sor:
>

Okay, its looks like Richardson/SOR solves this just fine. You can use this
as the smoother for GAMG instead
of Cheby/Jacobi, and probably see better results on the larger problems.

  Matt


> -ksp_type richardson
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type sor
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> Linear solve converged due to CONVERGED_RTOL iterations 40
>
> 3) And lastly with chebyshev/jacobi:
>
> -ksp_type chebyshev
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type jacobi
>
>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> Linear solve converged due to CONVERGED_RTOL iterations 67
>
> Looks like neither of these ksp/pc combos are good? I also tried
> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
> GAMG info I was able to grep from the out-of-box params:
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row
> (ave)=5, np=1
> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold 0.,
> 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
> min=2.394056e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 5.21674 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
> min=6.783380e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 9.85289 nnz ave. (N=537)
> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
> min=5.974776e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 12.4 nnz ave. (N=100)
> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
> min=4.842076e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>
> And the one with pc_gamg_agg_nsmooths 0
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row
> (ave)=5, np=1
> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold 0.,
> 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold 0.,
> 3.07398 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 3.02211 nnz ave. (N=814)
> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 3.00434 nnz ave. (N=461)
> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0., 3.
> nnz ave. (N=290)
> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0., 3.
> nnz ave. (N=197)
> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 2.98425 nnz ave. (N=127)
> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 2.97561 nnz ave. (N=82)
> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
> 2.9697 nnz ave. (N=66)
> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>
>
> Thanks,
> Justin
>
> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>>> Both GAMG and ILU are nice and dandy for this,
>>>
>>
>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
>> converge it way down, say rtol = 1.e-12. See which one is better in the
>> early iteration and pick it. It would be nice to check that it solves the
>> problem ...
>>
>> The residual drops about 5 orders in the first iteration and then
>> flatlines. That is very bad. Check that the smoother can actually solve the
>> problem.
>>
>>
>>> but as soon as I look at a bigger system, like a network with 8500
>>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>>> comes to tuning GAMG.
>>>
>>
>> First try -pc_gamg_nsmooths 0
>>
>> You can run with -info and grep on GAMG to see some diagnostic output.
>> Eigen estimates are fragile in practice and with this parameter and SOR
>> there are no eigen estimates needed. The max eigen values for all levels
>> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
>> problem.
>>
>>
>>> Attached is the ksp monitor/view output for gamg on the unsuccessful
>>> solve
>>>
>>> I'm also attaching a zip file which contains the simple PETSc script
>>> that loads the binary matrix/vector as well as two test cases, if you guys
>>> are interested in trying it out. It only works if you have PETSc configured
>>> with complex numbers.
>>>
>>> Thanks
>>>
>>> Justin
>>>
>>> PS - A couple years ago I had asked if there was a paper/tutorial on
>>> using/tuning GAMG. Does such a thing exist today?
>>>
>>
>> There is a write up in the manual that is tutorial like.
>>
>>
>>>
>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>>>> solve:
>>>>>
>>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>
>>>>> Right now we're just trying to solve eq(5) (in section 1), inverting
>>>>> the linear Y-bus matrix. Eventually we have to be able to solve equations
>>>>> like those in the next section.
>>>>>
>>>>
>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>> M-matrix to me (if all the y's are positive). This means
>>>> that it should be really easy to solve, and I think GAMG should do it.
>>>> You can start out just doing relaxation, like SOR, on
>>>> small examples.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm working with some folks to extract a linear system of equations
>>>>>>> from an external software package that solves power flow equations in
>>>>>>> complex form. Since that external package uses serial direct solvers like
>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>>>>>>> be solved in PETSc using its parallel solvers.
>>>>>>>
>>>>>>> I got mumps to achieve a very minor speedup across two MPI processes
>>>>>>> on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>>>>>>> seconds). However I want to use iterative solvers and preconditioners but I
>>>>>>> have never worked with complex numbers so I am not sure what the "best"
>>>>>>> options are given PETSc's capabilities.
>>>>>>>
>>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>>> form.
>>>>>>>
>>>>>>
>>>>>> 1) We really need to see the (simplified) equations
>>>>>>
>>>>>> 2) All complex equations can be converted to a system of real
>>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>>
>>>>>>  Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> Justin
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/dc5b451f/attachment-0001.html>

From knepley at gmail.com  Fri Feb  1 14:24:03 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 1 Feb 2019 15:24:03 -0500
Subject: [petsc-users] Reading a complex vector from HDF5
In-Reply-To: <CAOGsD9hZfbk28bKP+hi0bfXuJUiCSh3k1P3a1QJ4FyOP8pACrA@mail.gmail.com>
References: <CAOGsD9hZfbk28bKP+hi0bfXuJUiCSh3k1P3a1QJ4FyOP8pACrA@mail.gmail.com>
Message-ID: <CAMYG4GmjmUkph7cUbbRGURoMf6SO3aBbbYk8mkKrtOukai5DTQ@mail.gmail.com>

On Fri, Feb 1, 2019 at 3:21 PM Sajid Ali via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I'm trying to load a complex vector from hdf5 and I get and I get the
> following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: File contains real numbers but PETSc is configured for
> complex. The conversion is not yet implemented. Configure with
> --with-scalar-type=real.
>

When we save complex data in HDF5, we give it an extra dimension for the
two reals. This must be missing to get this message.

  Matt


> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision:
> ba892ce6f06cbab637e31ab2bf33fd996c92d19d  GIT Date: 2019-01-29 05:55:16
> +0100
> [0]PETSC ERROR: ./ex_modify on a  named xrm by sajid Fri Feb  1 14:18:30
> 2019
> [0]PETSC ERROR: Configure options
> --prefix=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/petsc-develop-gh5lollh4dzklppm7kb7wpjtyh4gi4mi
> --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0
> CFLAGS="-march=native -O2" FFLAGS="-march=native -O2"
> CXXFLAGS="-march=native -O2"
> --with-cc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpicc
> --with-cxx=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpic++
> --with-fc=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/mpich-3.3-x4tesuo4sxsomfxns5i26vco7ywojdmz/bin/mpif90
> --with-precision=double --with-scalar-type=complex
> --with-shared-libraries=1 --with-debugging=0 --with-64-bit-indices=0
> COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS=
> --with-blaslapack-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/openblas-0.3.5-heqggft2eii2xktdbyp2zovez2yj6ugd/lib/libopenblas.so
> --with-x=1 --with-clanguage=C --with-scalapack=0 --with-metis=1
> --with-metis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/metis-5.1.0-mugo6fn7u2nlz72vo4ajlhbba5q7a5jn
> --with-hdf5=1
> --with-hdf5-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/hdf5-1.10.4-pes54zx5fsbgdg2nr7reyopc4zfc43wo
> --with-hypre=0 --with-parmetis=1
> --with-parmetis-dir=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/parmetis-4.0.3-kizvrkoos4ddlfc7jsnatvh3mdk3lgvg
> --with-mumps=0 --with-trilinos=0 --with-cxx-dialect=C++11
> --with-superlu_dist-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/superlu-dist-develop-d6dqq3ribpkyoawnwuckbtydgddijj6j/include
> --with-superlu_dist-lib=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/superlu-dist-develop-d6dqq3ribpkyoawnwuckbtydgddijj6j/lib/libsuperlu_dist.a
> --with-superlu_dist=1 --with-suitesparse=0
> --with-zlib-include=/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/zlib-1.2.11-v4lhujkxdy6ukr5ymjxgxcfb2qoo6vf3/include
> --with-zlib-lib="-L/raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/zlib-1.2.11-v4lhujkxdy6ukr5ymjxgxcfb2qoo6vf3/lib
> -lz" --with-zlib=1
> [0]PETSC ERROR: #1 PetscViewerHDF5Load() line 1348 in
> /tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c
> [0]PETSC ERROR: #2 VecLoad_HDF5() line 170 in
> /tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #3 VecLoad_Default() line 288 in
> /tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #4 VecLoad() line 933 in
> /tmp/sajid/spack-stage/spack-stage-hy86Yb/petsc/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #5 main() line 132 in
> /raid/home/sajid/packages/xwp_petsc/2d/matter_repeat/ex_modify.c
> [0]PETSC ERROR: No PETSc Option Table entries
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
>
> As per src/vec/vec/examples/tutorials/ex10.c , a complex 1D vector is
> saves as a 2D real valued vector of shape dim,2. I've done the same with my
> data and yet I get the above error. Is there any special precaution I
> should take when making the HDF5 file ?
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/337e9728/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Fri Feb  1 14:38:34 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 1 Feb 2019 14:38:34 -0600
Subject: [petsc-users] Reading a complex vector from HDF5
In-Reply-To: <CAMYG4GmjmUkph7cUbbRGURoMf6SO3aBbbYk8mkKrtOukai5DTQ@mail.gmail.com>
References: <CAOGsD9hZfbk28bKP+hi0bfXuJUiCSh3k1P3a1QJ4FyOP8pACrA@mail.gmail.com>
	<CAMYG4GmjmUkph7cUbbRGURoMf6SO3aBbbYk8mkKrtOukai5DTQ@mail.gmail.com>
Message-ID: <CAOGsD9jmUO-yCF+xW9Y4gUJRcrgLtAGWweaAVRPHBWd8HQJ8PA@mail.gmail.com>

Column 1 contains the real value and column 2 contains the imaginary value,
correct?

I did that last time as well (and opened it using h5py just to be sure that
the shape is indeed dim x 2 and the datatype is f8),  yet I get the error.

The error comes from these lines in PETSc :

#if defined(PETSC_USE_COMPLEX)
if (!h->complexVal) {
H5T_class_t clazz = H5Tget_class(datatype);
if (clazz == H5T_FLOAT) SETERRQ(PetscObjectComm((PetscObject)viewer),
PETSC_ERR_SUP,"File contains real numbers but PETSc is configured for
complex. The conversion is not yet implemented. Configure with
--with-scalar-type=real.");
}

Am I setting the dtype incorrectly?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/a0c37fba/attachment.html>

From jychang48 at gmail.com  Fri Feb  1 14:53:21 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 1 Feb 2019 13:53:21 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
Message-ID: <CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>

I tried these options:

-ksp_view
-ksp_monitor_true_residual
-ksp_type gmres
-ksp_max_it 50
-pc_type gamg
-mg_coarse_pc_type sor
-mg_levels_1_ksp_type richardson
-mg_levels_2_ksp_type richardson
-mg_levels_3_ksp_type richardson
-mg_levels_4_ksp_type richardson
-mg_levels_1_pc_type sor
-mg_levels_2_pc_type sor
-mg_levels_3_pc_type sor
-mg_levels_4_pc_type sor

And still have a non-converging solution:

  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=50, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.
0.   0.
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
          Symmetric graph false
          Number of levels to square graph 1
          Number smoothing steps 1
        Complexity:    grid = 1.31821
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=17, cols=17
        total: nonzeros=163, allocated nonzeros=163
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 12 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 1 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=100, cols=100
        total: nonzeros=1240, allocated nonzeros=1240
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 1 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=537, cols=537
        total: nonzeros=5291, allocated nonzeros=5291
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 1 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=1541, cols=1541
        total: nonzeros=8039, allocated nonzeros=8039
        total number of mallocs used during MatSetValues calls =0
          not using I-node routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 1 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 1 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega
= 1.
      linear system matrix = precond matrix:
      Mat Object: 1 MPI processes
        type: seqaij
        rows=8541, cols=8541
        total: nonzeros=46299, allocated nonzeros=46299
        total number of mallocs used during MatSetValues calls =0
          using I-node routines: found 5464 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
    type: seqaij
    rows=8541, cols=8541
    total: nonzeros=46299, allocated nonzeros=46299
    total number of mallocs used during MatSetValues calls =0
      using I-node routines: found 5464 nodes, limit used is 5

Am I doing this right? Did I miss anything?



On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hi Mark,
>>
>> 1) So with these options:
>>
>> -ksp_type gmres
>> -ksp_rtol 1e-15
>> -ksp_monitor_true_residual
>> -ksp_converged_reason
>> -pc_type bjacobi
>>
>> This is what I get:
>>
>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>
>> 2) With richardson/sor:
>>
>
> Okay, its looks like Richardson/SOR solves this just fine. You can use
> this as the smoother for GAMG instead
> of Cheby/Jacobi, and probably see better results on the larger problems.
>
>   Matt
>
>
>> -ksp_type richardson
>> -ksp_rtol 1e-15
>> -ksp_monitor_true_residual
>> -pc_type sor
>>
>> This is what I get:
>>
>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>
>> 3) And lastly with chebyshev/jacobi:
>>
>> -ksp_type chebyshev
>> -ksp_rtol 1e-15
>> -ksp_monitor_true_residual
>> -pc_type jacobi
>>
>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>
>> Looks like neither of these ksp/pc combos are good? I also tried
>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
>> GAMG info I was able to grep from the out-of-box params:
>>
>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> nnz/row (ave)=5, np=1
>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>> 0., 5.42079 nnz ave. (N=8541)
>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>> min=2.394056e-02 PC=jacobi
>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
>> pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 5.21674 nnz ave. (N=1541)
>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>> min=6.783380e-02 PC=jacobi
>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 9.85289 nnz ave. (N=537)
>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>> min=5.974776e-02 PC=jacobi
>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
>> pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 12.4 nnz ave. (N=100)
>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>> min=4.842076e-01 PC=jacobi
>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>
>> And the one with pc_gamg_agg_nsmooths 0
>>
>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> nnz/row (ave)=5, np=1
>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>> 0., 5.42079 nnz ave. (N=8541)
>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold
>> 0., 3.07398 nnz ave. (N=1541)
>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 3.02211 nnz ave. (N=814)
>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 3.00434 nnz ave. (N=461)
>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 3. nnz ave. (N=290)
>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 3. nnz ave. (N=197)
>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 2.98425 nnz ave. (N=127)
>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 2.97561 nnz ave. (N=82)
>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>> 2.9697 nnz ave. (N=66)
>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>
>>
>> Thanks,
>> Justin
>>
>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>>
>>>
>>>> Both GAMG and ILU are nice and dandy for this,
>>>>
>>>
>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
>>> converge it way down, say rtol = 1.e-12. See which one is better in the
>>> early iteration and pick it. It would be nice to check that it solves the
>>> problem ...
>>>
>>> The residual drops about 5 orders in the first iteration and then
>>> flatlines. That is very bad. Check that the smoother can actually solve the
>>> problem.
>>>
>>>
>>>> but as soon as I look at a bigger system, like a network with 8500
>>>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>>>> comes to tuning GAMG.
>>>>
>>>
>>> First try -pc_gamg_nsmooths 0
>>>
>>> You can run with -info and grep on GAMG to see some diagnostic output.
>>> Eigen estimates are fragile in practice and with this parameter and SOR
>>> there are no eigen estimates needed. The max eigen values for all levels
>>> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
>>> problem.
>>>
>>>
>>>> Attached is the ksp monitor/view output for gamg on the unsuccessful
>>>> solve
>>>>
>>>> I'm also attaching a zip file which contains the simple PETSc script
>>>> that loads the binary matrix/vector as well as two test cases, if you guys
>>>> are interested in trying it out. It only works if you have PETSc configured
>>>> with complex numbers.
>>>>
>>>> Thanks
>>>>
>>>> Justin
>>>>
>>>> PS - A couple years ago I had asked if there was a paper/tutorial on
>>>> using/tuning GAMG. Does such a thing exist today?
>>>>
>>>
>>> There is a write up in the manual that is tutorial like.
>>>
>>>
>>>>
>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>>>>> solve:
>>>>>>
>>>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>>
>>>>>> Right now we're just trying to solve eq(5) (in section 1), inverting
>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve equations
>>>>>> like those in the next section.
>>>>>>
>>>>>
>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>>> M-matrix to me (if all the y's are positive). This means
>>>>> that it should be really easy to solve, and I think GAMG should do it.
>>>>> You can start out just doing relaxation, like SOR, on
>>>>> small examples.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm working with some folks to extract a linear system of equations
>>>>>>>> from an external software package that solves power flow equations in
>>>>>>>> complex form. Since that external package uses serial direct solvers like
>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>>>>>>>> be solved in PETSc using its parallel solvers.
>>>>>>>>
>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>>>>> processes on a single node (went from solving a 300k dog system in 1.8
>>>>>>>> seconds to 1.5 seconds). However I want to use iterative solvers and
>>>>>>>> preconditioners but I have never worked with complex numbers so I am not
>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>>>>>
>>>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>>>> form.
>>>>>>>>
>>>>>>>
>>>>>>> 1) We really need to see the (simplified) equations
>>>>>>>
>>>>>>> 2) All complex equations can be converted to a system of real
>>>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>>>
>>>>>>>  Thanks,
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>
>>>>>>>> Justin
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/c782f5ce/attachment-0001.html>

From huq2090 at gmail.com  Fri Feb  1 15:04:03 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Fri, 1 Feb 2019 15:04:03 -0600
Subject: [petsc-users] Problem in MPI execution
In-Reply-To: <5EB6EF10-BDFF-4B53-9C6F-8F2F8F68BE75@anl.gov>
References: <CAE5QvP0aT6MADLaEiwv5xNOs71D2D5Zw6SncyXw3aqpg7MCJ_Q@mail.gmail.com>
	<CAMYG4G=SQwiQZShWkXqiWkiYL1DebDYPQH-S7py5zeVGxu8nkQ@mail.gmail.com>
	<CAE5QvP0PxUdOMYCsRBvJN9iusT1-jNm9MLGHaFzbsf+2g=cx1Q@mail.gmail.com>
	<CAMYG4GkCmTrFhE6rJPksvE=JT1LF973NgMkT6TU1kaZUCSqoCw@mail.gmail.com>
	<5EB6EF10-BDFF-4B53-9C6F-8F2F8F68BE75@anl.gov>
Message-ID: <CAE5QvP1hoOA5D_8FvNt_kzv4=WSdXYx2XNHohQo+ef4YCTKd4g@mail.gmail.com>

Thanks!
I will try that.
Sincerely,
Huq

On Wed, Jan 30, 2019, 8:33 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>
> > On Jan 30, 2019, at 9:24 AM, Matthew Knepley via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > On Wed, Jan 30, 2019 at 8:57 AM Fazlul Huq <huq2090 at gmail.com> wrote:
> > Thanks Matt.
> >
> > Is there anyway to go over this problem?
> > I need to run program with parallel Cholesky and ILU.
> >
> > From the link I sent you, you can try MUMPS and PasTiX for Cholesky.
>
>    And look at the -pc_hypre_type option; one of them is for ILU.
>
>
> >
> >    Matt
> >
> > Thanks.
> > Sincerely,
> > Huq
> >
> > On Wed, Jan 30, 2019 at 7:32 AM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Wed, Jan 30, 2019 at 8:25 AM Fazlul Huq via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hello PETSc Developers,
> >
> > I am trying to run my code with the following commands:
> > $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 6 ./poisson_m -n $x -pc_type hypre
> -pc_hypre_type boomeramg
> > $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type
> cholesky -log_view
> > $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n 2 ./poisson_m -n $x -pc_type ilu
> -log_view
> >
> > We do not have parallel Cholesky or ILU by default. Here is the table of
> solvers:
> https://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
> >
> >   Matt
> >
> > It works with fine first case (-pc_type hypre -pc_hyper_type boomeramg)
> but for 2nd and 3rd case I got the following error message:
> >
> > Solving the problem...[0]PETSC ERROR: --------------------- Error
> Message --------------------------------------------------------------
> > [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
> possible LU and Cholesky solvers
> > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
> ./configure with --download-<package>
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> > [0]PETSC ERROR: ./poisson_m on a arch-linux2-c-debug named
> huq2090-XPS-15-9570 by huq2090 Wed Jan 30 07:17:40 2019
> > [0]PETSC ERROR: Configure options --with-debugging=no
> --with-64-bit-indices --download-hypre --download-mpich
> > [0]PETSC ERROR: #1 MatGetFactor() line 4415 in
> /home/huq2090/petsc-3.10.2/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
> /home/huq2090/petsc-3.10.2/src/ksp/pc/impls/factor/ilu/ilu.c
> > [0]PETSC ERROR: #3 PCSetUp() line 932 in
> /home/huq2090/petsc-3.10.2/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #4 KSPSetUp() line 391 in
> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #5 KSPSolve() line 723 in
> /home/huq2090/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #6 main() line 199 in
> /home/huq2090/petsc-3.10.2/problems/ksp/poisson_m.c
> > [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -log_view
> > [0]PETSC ERROR: -n 9999999
> > [0]PETSC ERROR: -pc_type ilu
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0
> >
> >
> > Any suggestion?
> > Thanks.
> > Sincerely,
> > Huq
> >
> > --
> >
> > Fazlul Huq
> > Graduate Research Assistant
> > Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> > University of Illinois at Urbana-Champaign (UIUC)
> > E-mail: huq2090 at gmail.com
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> >
> > Fazlul Huq
> > Graduate Research Assistant
> > Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> > University of Illinois at Urbana-Champaign (UIUC)
> > E-mail: huq2090 at gmail.com
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/0e423805/attachment.html>

From tisaac at cc.gatech.edu  Fri Feb  1 15:06:33 2019
From: tisaac at cc.gatech.edu (Isaac, Tobin G)
Date: Fri, 1 Feb 2019 21:06:33 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
Message-ID: <0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>

- Resuggesting a look at the max/min eigenvalues for as big a system as possible.
- I like doing fgmres with a monitored krylov method to see if the smoother is behaving very differently on the fine vs intermediate vs coarse levels


On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>I tried these options:
>
>-ksp_view
>-ksp_monitor_true_residual
>-ksp_type gmres
>-ksp_max_it 50
>-pc_type gamg
>-mg_coarse_pc_type sor
>-mg_levels_1_ksp_type richardson
>-mg_levels_2_ksp_type richardson
>-mg_levels_3_ksp_type richardson
>-mg_levels_4_ksp_type richardson
>-mg_levels_1_pc_type sor
>-mg_levels_2_pc_type sor
>-mg_levels_3_pc_type sor
>-mg_levels_4_pc_type sor
>
>And still have a non-converging solution:
>
>  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>KSP Object: 1 MPI processes
>  type: gmres
>restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
>with no iterative refinement
>    happy breakdown tolerance 1e-30
>  maximum iterations=50, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
>PC Object: 1 MPI processes
>  type: gamg
>    type is MULTIPLICATIVE, levels=5 cycles=v
>      Cycles per PCApply=1
>      Using externally compute Galerkin coarse grid matrices
>      GAMG specific options
>      Threshold for dropping small values in graph on each level =   0.
>0.   0.
>        Threshold scaling factor for each level not specified = 1.
>        AGG specific options
>          Symmetric graph false
>          Number of levels to square graph 1
>          Number smoothing steps 1
>        Complexity:    grid = 1.31821
>  Coarse grid solver -- level -------------------------------
>    KSP Object: (mg_coarse_) 1 MPI processes
>      type: preonly
>      maximum iterations=10000, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_coarse_) 1 MPI processes
>      type: sor
>    type = local_symmetric, iterations = 1, local iterations = 1, omega
>= 1.
>      linear system matrix = precond matrix:
>      Mat Object: 1 MPI processes
>        type: seqaij
>        rows=17, cols=17
>        total: nonzeros=163, allocated nonzeros=163
>        total number of mallocs used during MatSetValues calls =0
>          using I-node routines: found 12 nodes, limit used is 5
>  Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object: (mg_levels_1_) 1 MPI processes
>      type: richardson
>        damping factor=1.
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_1_) 1 MPI processes
>      type: sor
>    type = local_symmetric, iterations = 1, local iterations = 1, omega
>= 1.
>      linear system matrix = precond matrix:
>      Mat Object: 1 MPI processes
>        type: seqaij
>        rows=100, cols=100
>        total: nonzeros=1240, allocated nonzeros=1240
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 2 -------------------------------
>    KSP Object: (mg_levels_2_) 1 MPI processes
>      type: richardson
>        damping factor=1.
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_2_) 1 MPI processes
>      type: sor
>    type = local_symmetric, iterations = 1, local iterations = 1, omega
>= 1.
>      linear system matrix = precond matrix:
>      Mat Object: 1 MPI processes
>        type: seqaij
>        rows=537, cols=537
>        total: nonzeros=5291, allocated nonzeros=5291
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 3 -------------------------------
>    KSP Object: (mg_levels_3_) 1 MPI processes
>      type: richardson
>        damping factor=1.
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_3_) 1 MPI processes
>      type: sor
>    type = local_symmetric, iterations = 1, local iterations = 1, omega
>= 1.
>      linear system matrix = precond matrix:
>      Mat Object: 1 MPI processes
>        type: seqaij
>        rows=1541, cols=1541
>        total: nonzeros=8039, allocated nonzeros=8039
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 4 -------------------------------
>    KSP Object: (mg_levels_4_) 1 MPI processes
>      type: richardson
>        damping factor=1.
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_4_) 1 MPI processes
>      type: sor
>    type = local_symmetric, iterations = 1, local iterations = 1, omega
>= 1.
>      linear system matrix = precond matrix:
>      Mat Object: 1 MPI processes
>        type: seqaij
>        rows=8541, cols=8541
>        total: nonzeros=46299, allocated nonzeros=46299
>        total number of mallocs used during MatSetValues calls =0
>          using I-node routines: found 5464 nodes, limit used is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  linear system matrix = precond matrix:
>  Mat Object: 1 MPI processes
>    type: seqaij
>    rows=8541, cols=8541
>    total: nonzeros=46299, allocated nonzeros=46299
>    total number of mallocs used during MatSetValues calls =0
>      using I-node routines: found 5464 nodes, limit used is 5
>
>Am I doing this right? Did I miss anything?
>
>
>
>On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>wrote:
>
>> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>wrote:
>>
>>> Hi Mark,
>>>
>>> 1) So with these options:
>>>
>>> -ksp_type gmres
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -ksp_converged_reason
>>> -pc_type bjacobi
>>>
>>> This is what I get:
>>>
>>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>
>>> 2) With richardson/sor:
>>>
>>
>> Okay, its looks like Richardson/SOR solves this just fine. You can
>use
>> this as the smoother for GAMG instead
>> of Cheby/Jacobi, and probably see better results on the larger
>problems.
>>
>>   Matt
>>
>>
>>> -ksp_type richardson
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -pc_type sor
>>>
>>> This is what I get:
>>>
>>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>
>>> 3) And lastly with chebyshev/jacobi:
>>>
>>> -ksp_type chebyshev
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -pc_type jacobi
>>>
>>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>
>>> Looks like neither of these ksp/pc combos are good? I also tried
>>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
>Here's the
>>> GAMG info I was able to grep from the out-of-box params:
>>>
>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> nnz/row (ave)=5, np=1
>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>threshold
>>> 0., 5.42079 nnz ave. (N=8541)
>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>> min=2.394056e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 5.21674 nnz ave. (N=1541)
>>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>> min=6.783380e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 9.85289 nnz ave. (N=537)
>>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>> min=5.974776e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 12.4 nnz ave. (N=100)
>>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>> min=4.842076e-01 PC=jacobi
>>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
>active pes
>>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>
>>> And the one with pc_gamg_agg_nsmooths 0
>>>
>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> nnz/row (ave)=5, np=1
>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>threshold
>>> 0., 5.42079 nnz ave. (N=8541)
>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>active
>>> pes
>>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
>threshold
>>> 0., 3.07398 nnz ave. (N=1541)
>>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 3.02211 nnz ave. (N=814)
>>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 3.00434 nnz ave. (N=461)
>>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 3. nnz ave. (N=290)
>>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 3. nnz ave. (N=197)
>>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 2.98425 nnz ave. (N=127)
>>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 2.97561 nnz ave. (N=82)
>>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
>active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>0.,
>>> 2.9697 nnz ave. (N=66)
>>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
>active pes
>>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>
>>>
>>> Thanks,
>>> Justin
>>>
>>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>>
>>>>
>>>>> Both GAMG and ILU are nice and dandy for this,
>>>>>
>>>>
>>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>and
>>>> converge it way down, say rtol = 1.e-12. See which one is better in
>the
>>>> early iteration and pick it. It would be nice to check that it
>solves the
>>>> problem ...
>>>>
>>>> The residual drops about 5 orders in the first iteration and then
>>>> flatlines. That is very bad. Check that the smoother can actually
>solve the
>>>> problem.
>>>>
>>>>
>>>>> but as soon as I look at a bigger system, like a network with 8500
>>>>> buses, the out-of-box gamg craps out. I am not sure where to start
>when it
>>>>> comes to tuning GAMG.
>>>>>
>>>>
>>>> First try -pc_gamg_nsmooths 0
>>>>
>>>> You can run with -info and grep on GAMG to see some diagnostic
>output.
>>>> Eigen estimates are fragile in practice and with this parameter and
>SOR
>>>> there are no eigen estimates needed. The max eigen values for all
>levels
>>>> should be between say 2 (or less) and say 3-4. Much higher is a
>sign of a
>>>> problem.
>>>>
>>>>
>>>>> Attached is the ksp monitor/view output for gamg on the
>unsuccessful
>>>>> solve
>>>>>
>>>>> I'm also attaching a zip file which contains the simple PETSc
>script
>>>>> that loads the binary matrix/vector as well as two test cases, if
>you guys
>>>>> are interested in trying it out. It only works if you have PETSc
>configured
>>>>> with complex numbers.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Justin
>>>>>
>>>>> PS - A couple years ago I had asked if there was a paper/tutorial
>on
>>>>> using/tuning GAMG. Does such a thing exist today?
>>>>>
>>>>
>>>> There is a write up in the manual that is tutorial like.
>>>>
>>>>
>>>>>
>>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
><knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
><jychang48 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
>to
>>>>>>> solve:
>>>>>>>
>>>>>>>
>http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>>>
>>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>inverting
>>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
>equations
>>>>>>> like those in the next section.
>>>>>>>
>>>>>>
>>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>>>> M-matrix to me (if all the y's are positive). This means
>>>>>> that it should be really easy to solve, and I think GAMG should
>do it.
>>>>>> You can start out just doing relaxation, like SOR, on
>>>>>> small examples.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
><knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I'm working with some folks to extract a linear system of
>equations
>>>>>>>>> from an external software package that solves power flow
>equations in
>>>>>>>>> complex form. Since that external package uses serial direct
>solvers like
>>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same
>matrix can
>>>>>>>>> be solved in PETSc using its parallel solvers.
>>>>>>>>>
>>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>>>>>> processes on a single node (went from solving a 300k dog
>system in 1.8
>>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
>solvers and
>>>>>>>>> preconditioners but I have never worked with complex numbers
>so I am not
>>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>>>>>>
>>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
>(unsurprisingly). I
>>>>>>>>> believe I also tried BICG with BJACOBI and while it did
>converge it
>>>>>>>>> converged slowly. Does anyone have recommendations on how one
>would go
>>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
>was originally
>>>>>>>>> thinking about converting it to cartesian form: Declaring all
>voltages =
>>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a
>conditional
>>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
>literature that
>>>>>>>>> claim to successfully precondition power flow equations
>operate in this
>>>>>>>>> form.
>>>>>>>>>
>>>>>>>>
>>>>>>>> 1) We really need to see the (simplified) equations
>>>>>>>>
>>>>>>>> 2) All complex equations can be converted to a system of real
>>>>>>>> equations twice as large, but this is not necessarily the best
>way to go
>>>>>>>>
>>>>>>>>  Thanks,
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Justin
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin
>their
>>>>>>>> experiments is infinitely more interesting than any results to
>which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to
>which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>

From mfadams at lbl.gov  Fri Feb  1 15:18:10 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 1 Feb 2019 16:18:10 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
Message-ID: <CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>

We do need the equations and discretization, this not a Laplacian or
something is not working out-of-th-box.

For pc_gamg_agg_nsmooths 0 you want  -pc_gamg_square_graph 10

The info output snows the size of the systems on each level and the number
of non-zeros. with pc_gamg_agg_nsmooths 0 in 2D it courses very slowly. You
want a reduction factor between grids of say 4 - 10 between grids.

And here is shorthand for r/sor on all levels:

-mg_levels_ksp_type richardson
-mg_levels_pc_type sor

On Fri, Feb 1, 2019 at 4:07 PM Isaac, Tobin G via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> - Resuggesting a look at the max/min eigenvalues for as big a system as
> possible.
> - I like doing fgmres with a monitored krylov method to see if the
> smoother is behaving very differently on the fine vs intermediate vs coarse
> levels
>
>
> On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >I tried these options:
> >
> >-ksp_view
> >-ksp_monitor_true_residual
> >-ksp_type gmres
> >-ksp_max_it 50
> >-pc_type gamg
> >-mg_coarse_pc_type sor
> >-mg_levels_1_ksp_type richardson
> >-mg_levels_2_ksp_type richardson
> >-mg_levels_3_ksp_type richardson
> >-mg_levels_4_ksp_type richardson
> >-mg_levels_1_pc_type sor
> >-mg_levels_2_pc_type sor
> >-mg_levels_3_pc_type sor
> >-mg_levels_4_pc_type sor
> >
> >And still have a non-converging solution:
> >
> >  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
> >1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
> >  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
> >1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
> >  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
> >1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
> >  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
> >1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
> >  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
> >1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
> >  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
> >1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
> >  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
> >1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
> >  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
> >1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
> >  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
> >1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
> >  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
> >1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> > 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
> >1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> > 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
> >1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> > 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
> >1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> > 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
> >1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> > 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
> >1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> > 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
> >1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> > 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
> >1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> > 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
> >1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> > 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
> >1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> > 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
> >1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> > 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
> >1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> > 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
> >1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> > 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
> >1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> > 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
> >1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> > 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
> >1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> > 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
> >1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> > 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
> >1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> > 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
> >1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> > 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
> >1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> > 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
> >1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> > 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
> >1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> > 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
> >1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> > 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
> >1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> > 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
> >1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> > 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
> >1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> > 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
> >1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> > 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
> >1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> > 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
> >1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> > 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
> >1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> > 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
> >1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> > 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
> >1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> > 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
> >1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> > 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
> >1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> > 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
> >1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> > 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
> >1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> > 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
> >1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> > 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
> >1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> > 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
> >1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> > 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
> >1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> > 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
> >1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> > 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
> >1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> >KSP Object: 1 MPI processes
> >  type: gmres
> >restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
> >with no iterative refinement
> >    happy breakdown tolerance 1e-30
> >  maximum iterations=50, initial guess is zero
> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >  left preconditioning
> >  using PRECONDITIONED norm type for convergence test
> >PC Object: 1 MPI processes
> >  type: gamg
> >    type is MULTIPLICATIVE, levels=5 cycles=v
> >      Cycles per PCApply=1
> >      Using externally compute Galerkin coarse grid matrices
> >      GAMG specific options
> >      Threshold for dropping small values in graph on each level =   0.
> >0.   0.
> >        Threshold scaling factor for each level not specified = 1.
> >        AGG specific options
> >          Symmetric graph false
> >          Number of levels to square graph 1
> >          Number smoothing steps 1
> >        Complexity:    grid = 1.31821
> >  Coarse grid solver -- level -------------------------------
> >    KSP Object: (mg_coarse_) 1 MPI processes
> >      type: preonly
> >      maximum iterations=10000, initial guess is zero
> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >      left preconditioning
> >      using NONE norm type for convergence test
> >    PC Object: (mg_coarse_) 1 MPI processes
> >      type: sor
> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
> >= 1.
> >      linear system matrix = precond matrix:
> >      Mat Object: 1 MPI processes
> >        type: seqaij
> >        rows=17, cols=17
> >        total: nonzeros=163, allocated nonzeros=163
> >        total number of mallocs used during MatSetValues calls =0
> >          using I-node routines: found 12 nodes, limit used is 5
> >  Down solver (pre-smoother) on level 1 -------------------------------
> >    KSP Object: (mg_levels_1_) 1 MPI processes
> >      type: richardson
> >        damping factor=1.
> >      maximum iterations=2, nonzero initial guess
> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >      left preconditioning
> >      using NONE norm type for convergence test
> >    PC Object: (mg_levels_1_) 1 MPI processes
> >      type: sor
> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
> >= 1.
> >      linear system matrix = precond matrix:
> >      Mat Object: 1 MPI processes
> >        type: seqaij
> >        rows=100, cols=100
> >        total: nonzeros=1240, allocated nonzeros=1240
> >        total number of mallocs used during MatSetValues calls =0
> >          not using I-node routines
> >  Up solver (post-smoother) same as down solver (pre-smoother)
> >  Down solver (pre-smoother) on level 2 -------------------------------
> >    KSP Object: (mg_levels_2_) 1 MPI processes
> >      type: richardson
> >        damping factor=1.
> >      maximum iterations=2, nonzero initial guess
> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >      left preconditioning
> >      using NONE norm type for convergence test
> >    PC Object: (mg_levels_2_) 1 MPI processes
> >      type: sor
> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
> >= 1.
> >      linear system matrix = precond matrix:
> >      Mat Object: 1 MPI processes
> >        type: seqaij
> >        rows=537, cols=537
> >        total: nonzeros=5291, allocated nonzeros=5291
> >        total number of mallocs used during MatSetValues calls =0
> >          not using I-node routines
> >  Up solver (post-smoother) same as down solver (pre-smoother)
> >  Down solver (pre-smoother) on level 3 -------------------------------
> >    KSP Object: (mg_levels_3_) 1 MPI processes
> >      type: richardson
> >        damping factor=1.
> >      maximum iterations=2, nonzero initial guess
> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >      left preconditioning
> >      using NONE norm type for convergence test
> >    PC Object: (mg_levels_3_) 1 MPI processes
> >      type: sor
> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
> >= 1.
> >      linear system matrix = precond matrix:
> >      Mat Object: 1 MPI processes
> >        type: seqaij
> >        rows=1541, cols=1541
> >        total: nonzeros=8039, allocated nonzeros=8039
> >        total number of mallocs used during MatSetValues calls =0
> >          not using I-node routines
> >  Up solver (post-smoother) same as down solver (pre-smoother)
> >  Down solver (pre-smoother) on level 4 -------------------------------
> >    KSP Object: (mg_levels_4_) 1 MPI processes
> >      type: richardson
> >        damping factor=1.
> >      maximum iterations=2, nonzero initial guess
> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >      left preconditioning
> >      using NONE norm type for convergence test
> >    PC Object: (mg_levels_4_) 1 MPI processes
> >      type: sor
> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
> >= 1.
> >      linear system matrix = precond matrix:
> >      Mat Object: 1 MPI processes
> >        type: seqaij
> >        rows=8541, cols=8541
> >        total: nonzeros=46299, allocated nonzeros=46299
> >        total number of mallocs used during MatSetValues calls =0
> >          using I-node routines: found 5464 nodes, limit used is 5
> >  Up solver (post-smoother) same as down solver (pre-smoother)
> >  linear system matrix = precond matrix:
> >  Mat Object: 1 MPI processes
> >    type: seqaij
> >    rows=8541, cols=8541
> >    total: nonzeros=46299, allocated nonzeros=46299
> >    total number of mallocs used during MatSetValues calls =0
> >      using I-node routines: found 5464 nodes, limit used is 5
> >
> >Am I doing this right? Did I miss anything?
> >
> >
> >
> >On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
> >wrote:
> >
> >> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
> >wrote:
> >>
> >>> Hi Mark,
> >>>
> >>> 1) So with these options:
> >>>
> >>> -ksp_type gmres
> >>> -ksp_rtol 1e-15
> >>> -ksp_monitor_true_residual
> >>> -ksp_converged_reason
> >>> -pc_type bjacobi
> >>>
> >>> This is what I get:
> >>>
> >>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
> >>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
> >>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
> >>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
> >>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
> >>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
> >>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
> >>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
> >>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
> >>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> >>> Linear solve converged due to CONVERGED_RTOL iterations 5
> >>>
> >>> 2) With richardson/sor:
> >>>
> >>
> >> Okay, its looks like Richardson/SOR solves this just fine. You can
> >use
> >> this as the smoother for GAMG instead
> >> of Cheby/Jacobi, and probably see better results on the larger
> >problems.
> >>
> >>   Matt
> >>
> >>
> >>> -ksp_type richardson
> >>> -ksp_rtol 1e-15
> >>> -ksp_monitor_true_residual
> >>> -pc_type sor
> >>>
> >>> This is what I get:
> >>>
> >>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
> >>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
> >>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
> >>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
> >>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
> >>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
> >>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
> >>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
> >>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
> >>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
> >>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
> >>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
> >>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
> >>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
> >>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
> >>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
> >>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
> >>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
> >>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
> >>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
> >>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
> >>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
> >>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
> >>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
> >>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
> >>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
> >>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
> >>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
> >>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
> >>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
> >>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
> >>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
> >>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
> >>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
> >>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
> >>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
> >>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
> >>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
> >>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
> >>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
> >>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
> >>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
> >>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
> >>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
> >>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
> >>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
> >>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
> >>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
> >>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
> >>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
> >>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
> >>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
> >>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
> >>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
> >>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
> >>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
> >>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
> >>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
> >>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
> >>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
> >>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
> >>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
> >>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
> >>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
> >>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
> >>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
> >>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
> >>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
> >>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
> >>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
> >>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
> >>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
> >>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
> >>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
> >>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
> >>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
> >>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
> >>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
> >>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
> >>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> >>> Linear solve converged due to CONVERGED_RTOL iterations 40
> >>>
> >>> 3) And lastly with chebyshev/jacobi:
> >>>
> >>> -ksp_type chebyshev
> >>> -ksp_rtol 1e-15
> >>> -ksp_monitor_true_residual
> >>> -pc_type jacobi
> >>>
> >>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
> >>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
> >>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
> >>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
> >>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
> >>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
> >>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
> >>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
> >>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
> >>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
> >>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
> >>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
> >>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
> >>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
> >>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
> >>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
> >>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
> >>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
> >>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
> >>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
> >>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
> >>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
> >>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
> >>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
> >>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
> >>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
> >>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
> >>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
> >>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
> >>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
> >>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
> >>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
> >>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
> >>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
> >>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
> >>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
> >>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
> >>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
> >>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
> >>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
> >>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
> >>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
> >>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
> >>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
> >>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
> >>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
> >>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
> >>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
> >>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
> >>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
> >>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
> >>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
> >>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
> >>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
> >>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
> >>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
> >>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
> >>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
> >>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
> >>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
> >>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
> >>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
> >>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
> >>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
> >>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
> >>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
> >>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
> >>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
> >>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
> >>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
> >>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
> >>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
> >>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
> >>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
> >>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
> >>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
> >>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
> >>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
> >>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
> >>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
> >>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
> >>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
> >>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
> >>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
> >>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
> >>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
> >>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
> >>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
> >>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
> >>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
> >>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
> >>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
> >>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
> >>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
> >>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
> >>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
> >>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
> >>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
> >>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
> >>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
> >>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
> >>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
> >>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
> >>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
> >>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
> >>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
> >>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
> >>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
> >>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
> >>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
> >>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
> >>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
> >>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
> >>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
> >>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
> >>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
> >>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
> >>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
> >>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
> >>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
> >>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
> >>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
> >>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
> >>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
> >>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
> >>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
> >>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
> >>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
> >>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
> >>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
> >>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
> >>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
> >>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
> >>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> >>> Linear solve converged due to CONVERGED_RTOL iterations 67
> >>>
> >>> Looks like neither of these ksp/pc combos are good? I also tried
> >>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
> >Here's the
> >>> GAMG info I was able to grep from the out-of-box params:
> >>>
> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> >>> nnz/row (ave)=5, np=1
> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
> >threshold
> >>> 0., 5.42079 nnz ave. (N=8541)
> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
> >>> min=2.394056e-02 PC=jacobi
> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
> >active
> >>> pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 5.21674 nnz ave. (N=1541)
> >>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
> >>> min=6.783380e-02 PC=jacobi
> >>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 9.85289 nnz ave. (N=537)
> >>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
> >>> min=5.974776e-02 PC=jacobi
> >>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
> >active
> >>> pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 12.4 nnz ave. (N=100)
> >>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
> >>> min=4.842076e-01 PC=jacobi
> >>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
> >active pes
> >>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> >>>
> >>> And the one with pc_gamg_agg_nsmooths 0
> >>>
> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> >>> nnz/row (ave)=5, np=1
> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
> >threshold
> >>> 0., 5.42079 nnz ave. (N=8541)
> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
> >active
> >>> pes
> >>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
> >threshold
> >>> 0., 3.07398 nnz ave. (N=1541)
> >>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> >>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 3.02211 nnz ave. (N=814)
> >>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> >>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 3.00434 nnz ave. (N=461)
> >>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> >>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 3. nnz ave. (N=290)
> >>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> >>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 3. nnz ave. (N=197)
> >>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> >>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 2.98425 nnz ave. (N=127)
> >>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> >>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 2.97561 nnz ave. (N=82)
> >>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> >>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
> >active pes
> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
> >0.,
> >>> 2.9697 nnz ave. (N=66)
> >>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> >>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
> >active pes
> >>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> >>>
> >>>
> >>> Thanks,
> >>> Justin
> >>>
> >>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> >>>
> >>>>
> >>>>
> >>>>> Both GAMG and ILU are nice and dandy for this,
> >>>>>
> >>>>
> >>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
> >and
> >>>> converge it way down, say rtol = 1.e-12. See which one is better in
> >the
> >>>> early iteration and pick it. It would be nice to check that it
> >solves the
> >>>> problem ...
> >>>>
> >>>> The residual drops about 5 orders in the first iteration and then
> >>>> flatlines. That is very bad. Check that the smoother can actually
> >solve the
> >>>> problem.
> >>>>
> >>>>
> >>>>> but as soon as I look at a bigger system, like a network with 8500
> >>>>> buses, the out-of-box gamg craps out. I am not sure where to start
> >when it
> >>>>> comes to tuning GAMG.
> >>>>>
> >>>>
> >>>> First try -pc_gamg_nsmooths 0
> >>>>
> >>>> You can run with -info and grep on GAMG to see some diagnostic
> >output.
> >>>> Eigen estimates are fragile in practice and with this parameter and
> >SOR
> >>>> there are no eigen estimates needed. The max eigen values for all
> >levels
> >>>> should be between say 2 (or less) and say 3-4. Much higher is a
> >sign of a
> >>>> problem.
> >>>>
> >>>>
> >>>>> Attached is the ksp monitor/view output for gamg on the
> >unsuccessful
> >>>>> solve
> >>>>>
> >>>>> I'm also attaching a zip file which contains the simple PETSc
> >script
> >>>>> that loads the binary matrix/vector as well as two test cases, if
> >you guys
> >>>>> are interested in trying it out. It only works if you have PETSc
> >configured
> >>>>> with complex numbers.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Justin
> >>>>>
> >>>>> PS - A couple years ago I had asked if there was a paper/tutorial
> >on
> >>>>> using/tuning GAMG. Does such a thing exist today?
> >>>>>
> >>>>
> >>>> There is a write up in the manual that is tutorial like.
> >>>>
> >>>>
> >>>>>
> >>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
> ><knepley at gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
> ><jychang48 at gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
> >to
> >>>>>>> solve:
> >>>>>>>
> >>>>>>>
> >http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> >>>>>>>
> >>>>>>> Right now we're just trying to solve eq(5) (in section 1),
> >inverting
> >>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
> >equations
> >>>>>>> like those in the next section.
> >>>>>>>
> >>>>>>
> >>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
> >>>>>> M-matrix to me (if all the y's are positive). This means
> >>>>>> that it should be really easy to solve, and I think GAMG should
> >do it.
> >>>>>> You can start out just doing relaxation, like SOR, on
> >>>>>> small examples.
> >>>>>>
> >>>>>>   Thanks,
> >>>>>>
> >>>>>>     Matt
> >>>>>>
> >>>>>>
> >>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
> ><knepley at gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
> >>>>>>>> petsc-users at mcs.anl.gov> wrote:
> >>>>>>>>
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> I'm working with some folks to extract a linear system of
> >equations
> >>>>>>>>> from an external software package that solves power flow
> >equations in
> >>>>>>>>> complex form. Since that external package uses serial direct
> >solvers like
> >>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same
> >matrix can
> >>>>>>>>> be solved in PETSc using its parallel solvers.
> >>>>>>>>>
> >>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
> >>>>>>>>> processes on a single node (went from solving a 300k dog
> >system in 1.8
> >>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
> >solvers and
> >>>>>>>>> preconditioners but I have never worked with complex numbers
> >so I am not
> >>>>>>>>> sure what the "best" options are given PETSc's capabilities.
> >>>>>>>>>
> >>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
> >(unsurprisingly). I
> >>>>>>>>> believe I also tried BICG with BJACOBI and while it did
> >converge it
> >>>>>>>>> converged slowly. Does anyone have recommendations on how one
> >would go
> >>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
> >was originally
> >>>>>>>>> thinking about converting it to cartesian form: Declaring all
> >voltages =
> >>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a
> >conditional
> >>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
> >literature that
> >>>>>>>>> claim to successfully precondition power flow equations
> >operate in this
> >>>>>>>>> form.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> 1) We really need to see the (simplified) equations
> >>>>>>>>
> >>>>>>>> 2) All complex equations can be converted to a system of real
> >>>>>>>> equations twice as large, but this is not necessarily the best
> >way to go
> >>>>>>>>
> >>>>>>>>  Thanks,
> >>>>>>>>
> >>>>>>>>     Matt
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Justin
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> What most experimenters take for granted before they begin
> >their
> >>>>>>>> experiments is infinitely more interesting than any results to
> >which their
> >>>>>>>> experiments lead.
> >>>>>>>> -- Norbert Wiener
> >>>>>>>>
> >>>>>>>> https://www.cse.buffalo.edu/~knepley/
> >>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> What most experimenters take for granted before they begin their
> >>>>>> experiments is infinitely more interesting than any results to
> >which their
> >>>>>> experiments lead.
> >>>>>> -- Norbert Wiener
> >>>>>>
> >>>>>> https://www.cse.buffalo.edu/~knepley/
> >>>>>> <http://www.cse.buffalo.edu/~knepley/>
> >>>>>>
> >>>>>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> >> experiments is infinitely more interesting than any results to which
> >their
> >> experiments lead.
> >> -- Norbert Wiener
> >>
> >> https://www.cse.buffalo.edu/~knepley/
> >> <http://www.cse.buffalo.edu/~knepley/>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/68452957/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Fri Feb  1 15:52:47 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 1 Feb 2019 15:52:47 -0600
Subject: [petsc-users] Reading a complex vector from HDF5
In-Reply-To: <CAOGsD9jmUO-yCF+xW9Y4gUJRcrgLtAGWweaAVRPHBWd8HQJ8PA@mail.gmail.com>
References: <CAOGsD9hZfbk28bKP+hi0bfXuJUiCSh3k1P3a1QJ4FyOP8pACrA@mail.gmail.com>
	<CAMYG4GmjmUkph7cUbbRGURoMf6SO3aBbbYk8mkKrtOukai5DTQ@mail.gmail.com>
	<CAOGsD9jmUO-yCF+xW9Y4gUJRcrgLtAGWweaAVRPHBWd8HQJ8PA@mail.gmail.com>
Message-ID: <CAOGsD9ikPjLYrC_D7WHQVwaeCG5xjSR59LrroPzQR4CmtuWMJQ@mail.gmail.com>

I think I understand what's happening here, when I look at a data file I
create similar to the aforementioned example, I see a complex=1 attribute
that I'm missing when I make my hdf5 file.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/9a23f990/attachment.html>

From dave.mayhem23 at gmail.com  Fri Feb  1 16:32:24 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Fri, 1 Feb 2019 22:32:24 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
	<CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>
Message-ID: <CAJ98EDqeogf3pLb00KLw5z-CL03SKiX8U-SCZZdp992NQr9LSA@mail.gmail.com>

I'd suggest
1) using LU on the coarse grid (you have 1 application of SOR) and
2) do more Richardson iterations on each level (you have the default of 2).
Try 4 or 6

Thanks,
  Dave

On Fri, 1 Feb 2019 at 22:18, Mark Adams via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> We do need the equations and discretization, this not a Laplacian or
> something is not working out-of-th-box.
>
> For pc_gamg_agg_nsmooths 0 you want  -pc_gamg_square_graph 10
>
> The info output snows the size of the systems on each level and the number
> of non-zeros. with pc_gamg_agg_nsmooths 0 in 2D it courses very slowly. You
> want a reduction factor between grids of say 4 - 10 between grids.
>
> And here is shorthand for r/sor on all levels:
>
> -mg_levels_ksp_type richardson
> -mg_levels_pc_type sor
>
> On Fri, Feb 1, 2019 at 4:07 PM Isaac, Tobin G via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> - Resuggesting a look at the max/min eigenvalues for as big a system as
>> possible.
>> - I like doing fgmres with a monitored krylov method to see if the
>> smoother is behaving very differently on the fine vs intermediate vs coarse
>> levels
>>
>>
>> On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >I tried these options:
>> >
>> >-ksp_view
>> >-ksp_monitor_true_residual
>> >-ksp_type gmres
>> >-ksp_max_it 50
>> >-pc_type gamg
>> >-mg_coarse_pc_type sor
>> >-mg_levels_1_ksp_type richardson
>> >-mg_levels_2_ksp_type richardson
>> >-mg_levels_3_ksp_type richardson
>> >-mg_levels_4_ksp_type richardson
>> >-mg_levels_1_pc_type sor
>> >-mg_levels_2_pc_type sor
>> >-mg_levels_3_pc_type sor
>> >-mg_levels_4_pc_type sor
>> >
>> >And still have a non-converging solution:
>> >
>> >  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>> >1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>> >  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>> >1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>> >  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>> >1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>> >  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>> >1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>> >  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>> >1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>> >  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>> >1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>> >  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>> >1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>> >  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>> >1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>> >  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>> >1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>> >  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>> >1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>> > 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>> >1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>> > 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>> >1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>> > 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>> >1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>> > 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>> >1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>> > 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>> >1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>> > 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>> >1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>> > 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>> >1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>> > 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>> >1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>> > 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>> >1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>> > 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>> >1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>> > 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>> >1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>> > 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>> >1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>> > 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>> >1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>> > 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>> >1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>> > 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>> >1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>> > 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>> >1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>> > 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>> >1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>> > 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>> >1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>> > 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>> >1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>> > 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>> >1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>> > 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>> >1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>> > 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>> >1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>> > 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>> >1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>> > 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>> >1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>> > 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>> >1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>> > 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>> >1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>> > 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>> >1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>> > 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>> >1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>> > 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>> >1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>> > 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>> >1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>> > 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>> >1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>> > 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>> >1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>> > 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>> >1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>> > 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>> >1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>> > 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>> >1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>> > 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>> >1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>> > 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>> >1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>> > 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>> >1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>> > 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>> >1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>> > 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>> >1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>> > 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>> >1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>> >KSP Object: 1 MPI processes
>> >  type: gmres
>> >restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
>> >with no iterative refinement
>> >    happy breakdown tolerance 1e-30
>> >  maximum iterations=50, initial guess is zero
>> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >  left preconditioning
>> >  using PRECONDITIONED norm type for convergence test
>> >PC Object: 1 MPI processes
>> >  type: gamg
>> >    type is MULTIPLICATIVE, levels=5 cycles=v
>> >      Cycles per PCApply=1
>> >      Using externally compute Galerkin coarse grid matrices
>> >      GAMG specific options
>> >      Threshold for dropping small values in graph on each level =   0.
>> >0.   0.
>> >        Threshold scaling factor for each level not specified = 1.
>> >        AGG specific options
>> >          Symmetric graph false
>> >          Number of levels to square graph 1
>> >          Number smoothing steps 1
>> >        Complexity:    grid = 1.31821
>> >  Coarse grid solver -- level -------------------------------
>> >    KSP Object: (mg_coarse_) 1 MPI processes
>> >      type: preonly
>> >      maximum iterations=10000, initial guess is zero
>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >      left preconditioning
>> >      using NONE norm type for convergence test
>> >    PC Object: (mg_coarse_) 1 MPI processes
>> >      type: sor
>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>> >= 1.
>> >      linear system matrix = precond matrix:
>> >      Mat Object: 1 MPI processes
>> >        type: seqaij
>> >        rows=17, cols=17
>> >        total: nonzeros=163, allocated nonzeros=163
>> >        total number of mallocs used during MatSetValues calls =0
>> >          using I-node routines: found 12 nodes, limit used is 5
>> >  Down solver (pre-smoother) on level 1 -------------------------------
>> >    KSP Object: (mg_levels_1_) 1 MPI processes
>> >      type: richardson
>> >        damping factor=1.
>> >      maximum iterations=2, nonzero initial guess
>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >      left preconditioning
>> >      using NONE norm type for convergence test
>> >    PC Object: (mg_levels_1_) 1 MPI processes
>> >      type: sor
>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>> >= 1.
>> >      linear system matrix = precond matrix:
>> >      Mat Object: 1 MPI processes
>> >        type: seqaij
>> >        rows=100, cols=100
>> >        total: nonzeros=1240, allocated nonzeros=1240
>> >        total number of mallocs used during MatSetValues calls =0
>> >          not using I-node routines
>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>> >  Down solver (pre-smoother) on level 2 -------------------------------
>> >    KSP Object: (mg_levels_2_) 1 MPI processes
>> >      type: richardson
>> >        damping factor=1.
>> >      maximum iterations=2, nonzero initial guess
>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >      left preconditioning
>> >      using NONE norm type for convergence test
>> >    PC Object: (mg_levels_2_) 1 MPI processes
>> >      type: sor
>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>> >= 1.
>> >      linear system matrix = precond matrix:
>> >      Mat Object: 1 MPI processes
>> >        type: seqaij
>> >        rows=537, cols=537
>> >        total: nonzeros=5291, allocated nonzeros=5291
>> >        total number of mallocs used during MatSetValues calls =0
>> >          not using I-node routines
>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>> >  Down solver (pre-smoother) on level 3 -------------------------------
>> >    KSP Object: (mg_levels_3_) 1 MPI processes
>> >      type: richardson
>> >        damping factor=1.
>> >      maximum iterations=2, nonzero initial guess
>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >      left preconditioning
>> >      using NONE norm type for convergence test
>> >    PC Object: (mg_levels_3_) 1 MPI processes
>> >      type: sor
>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>> >= 1.
>> >      linear system matrix = precond matrix:
>> >      Mat Object: 1 MPI processes
>> >        type: seqaij
>> >        rows=1541, cols=1541
>> >        total: nonzeros=8039, allocated nonzeros=8039
>> >        total number of mallocs used during MatSetValues calls =0
>> >          not using I-node routines
>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>> >  Down solver (pre-smoother) on level 4 -------------------------------
>> >    KSP Object: (mg_levels_4_) 1 MPI processes
>> >      type: richardson
>> >        damping factor=1.
>> >      maximum iterations=2, nonzero initial guess
>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >      left preconditioning
>> >      using NONE norm type for convergence test
>> >    PC Object: (mg_levels_4_) 1 MPI processes
>> >      type: sor
>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>> >= 1.
>> >      linear system matrix = precond matrix:
>> >      Mat Object: 1 MPI processes
>> >        type: seqaij
>> >        rows=8541, cols=8541
>> >        total: nonzeros=46299, allocated nonzeros=46299
>> >        total number of mallocs used during MatSetValues calls =0
>> >          using I-node routines: found 5464 nodes, limit used is 5
>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>> >  linear system matrix = precond matrix:
>> >  Mat Object: 1 MPI processes
>> >    type: seqaij
>> >    rows=8541, cols=8541
>> >    total: nonzeros=46299, allocated nonzeros=46299
>> >    total number of mallocs used during MatSetValues calls =0
>> >      using I-node routines: found 5464 nodes, limit used is 5
>> >
>> >Am I doing this right? Did I miss anything?
>> >
>> >
>> >
>> >On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>> >wrote:
>> >
>> >> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>> >wrote:
>> >>
>> >>> Hi Mark,
>> >>>
>> >>> 1) So with these options:
>> >>>
>> >>> -ksp_type gmres
>> >>> -ksp_rtol 1e-15
>> >>> -ksp_monitor_true_residual
>> >>> -ksp_converged_reason
>> >>> -pc_type bjacobi
>> >>>
>> >>> This is what I get:
>> >>>
>> >>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>> >>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>> >>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>> >>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>> >>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>> >>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>> >>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>> >>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>> >>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>> >>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>> >>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>> >>> Linear solve converged due to CONVERGED_RTOL iterations 5
>> >>>
>> >>> 2) With richardson/sor:
>> >>>
>> >>
>> >> Okay, its looks like Richardson/SOR solves this just fine. You can
>> >use
>> >> this as the smoother for GAMG instead
>> >> of Cheby/Jacobi, and probably see better results on the larger
>> >problems.
>> >>
>> >>   Matt
>> >>
>> >>
>> >>> -ksp_type richardson
>> >>> -ksp_rtol 1e-15
>> >>> -ksp_monitor_true_residual
>> >>> -pc_type sor
>> >>>
>> >>> This is what I get:
>> >>>
>> >>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>> >>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>> >>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>> >>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>> >>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>> >>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>> >>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>> >>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>> >>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>> >>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>> >>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>> >>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>> >>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>> >>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>> >>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>> >>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>> >>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>> >>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>> >>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>> >>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>> >>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>> >>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>> >>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>> >>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>> >>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>> >>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>> >>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>> >>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>> >>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>> >>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>> >>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>> >>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>> >>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>> >>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>> >>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>> >>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>> >>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>> >>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>> >>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>> >>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>> >>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>> >>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>> >>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>> >>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>> >>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>> >>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>> >>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>> >>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>> >>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>> >>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>> >>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>> >>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>> >>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>> >>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>> >>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>> >>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>> >>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>> >>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>> >>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>> >>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>> >>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>> >>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>> >>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>> >>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>> >>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>> >>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>> >>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>> >>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>> >>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>> >>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>> >>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>> >>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>> >>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>> >>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>> >>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>> >>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>> >>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>> >>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>> >>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>> >>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>> >>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>> >>> Linear solve converged due to CONVERGED_RTOL iterations 40
>> >>>
>> >>> 3) And lastly with chebyshev/jacobi:
>> >>>
>> >>> -ksp_type chebyshev
>> >>> -ksp_rtol 1e-15
>> >>> -ksp_monitor_true_residual
>> >>> -pc_type jacobi
>> >>>
>> >>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>> >>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>> >>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>> >>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>> >>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>> >>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>> >>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>> >>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>> >>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>> >>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>> >>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>> >>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>> >>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>> >>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>> >>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>> >>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>> >>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>> >>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>> >>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>> >>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>> >>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>> >>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>> >>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>> >>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>> >>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>> >>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>> >>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>> >>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>> >>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>> >>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>> >>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>> >>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>> >>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>> >>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>> >>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>> >>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>> >>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>> >>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>> >>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>> >>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>> >>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>> >>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>> >>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>> >>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>> >>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>> >>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>> >>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>> >>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>> >>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>> >>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>> >>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>> >>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>> >>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>> >>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>> >>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>> >>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>> >>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>> >>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>> >>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>> >>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>> >>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>> >>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>> >>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>> >>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>> >>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>> >>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>> >>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>> >>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>> >>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>> >>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>> >>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>> >>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>> >>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>> >>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>> >>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>> >>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>> >>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>> >>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>> >>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>> >>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>> >>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>> >>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>> >>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>> >>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>> >>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>> >>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>> >>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>> >>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>> >>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>> >>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>> >>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>> >>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>> >>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>> >>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>> >>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>> >>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>> >>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>> >>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>> >>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>> >>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>> >>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>> >>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>> >>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>> >>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>> >>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>> >>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>> >>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>> >>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>> >>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>> >>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>> >>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>> >>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>> >>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>> >>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>> >>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>> >>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>> >>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>> >>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>> >>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>> >>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>> >>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>> >>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>> >>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>> >>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>> >>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>> >>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>> >>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>> >>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>> >>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>> >>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>> >>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>> >>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>> >>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>> >>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>> >>> Linear solve converged due to CONVERGED_RTOL iterations 67
>> >>>
>> >>> Looks like neither of these ksp/pc combos are good? I also tried
>> >>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
>> >Here's the
>> >>> GAMG info I was able to grep from the out-of-box params:
>> >>>
>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> >>> nnz/row (ave)=5, np=1
>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>> >threshold
>> >>> 0., 5.42079 nnz ave. (N=8541)
>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>> >>> min=2.394056e-02 PC=jacobi
>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>> >active
>> >>> pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 5.21674 nnz ave. (N=1541)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>> >>> min=6.783380e-02 PC=jacobi
>> >>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 9.85289 nnz ave. (N=537)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>> >>> min=5.974776e-02 PC=jacobi
>> >>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>> >active
>> >>> pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 12.4 nnz ave. (N=100)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>> >>> min=4.842076e-01 PC=jacobi
>> >>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
>> >active pes
>> >>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>> >>>
>> >>> And the one with pc_gamg_agg_nsmooths 0
>> >>>
>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> >>> nnz/row (ave)=5, np=1
>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>> >threshold
>> >>> 0., 5.42079 nnz ave. (N=8541)
>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>> >active
>> >>> pes
>> >>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
>> >threshold
>> >>> 0., 3.07398 nnz ave. (N=1541)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>> >>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 3.02211 nnz ave. (N=814)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>> >>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 3.00434 nnz ave. (N=461)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>> >>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 3. nnz ave. (N=290)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>> >>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 3. nnz ave. (N=197)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>> >>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 2.98425 nnz ave. (N=127)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>> >>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 2.97561 nnz ave. (N=82)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>> >>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
>> >active pes
>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>> >0.,
>> >>> 2.9697 nnz ave. (N=66)
>> >>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>> >>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
>> >active pes
>> >>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>> >>>
>> >>>
>> >>> Thanks,
>> >>> Justin
>> >>>
>> >>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>>> Both GAMG and ILU are nice and dandy for this,
>> >>>>>
>> >>>>
>> >>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>> >and
>> >>>> converge it way down, say rtol = 1.e-12. See which one is better in
>> >the
>> >>>> early iteration and pick it. It would be nice to check that it
>> >solves the
>> >>>> problem ...
>> >>>>
>> >>>> The residual drops about 5 orders in the first iteration and then
>> >>>> flatlines. That is very bad. Check that the smoother can actually
>> >solve the
>> >>>> problem.
>> >>>>
>> >>>>
>> >>>>> but as soon as I look at a bigger system, like a network with 8500
>> >>>>> buses, the out-of-box gamg craps out. I am not sure where to start
>> >when it
>> >>>>> comes to tuning GAMG.
>> >>>>>
>> >>>>
>> >>>> First try -pc_gamg_nsmooths 0
>> >>>>
>> >>>> You can run with -info and grep on GAMG to see some diagnostic
>> >output.
>> >>>> Eigen estimates are fragile in practice and with this parameter and
>> >SOR
>> >>>> there are no eigen estimates needed. The max eigen values for all
>> >levels
>> >>>> should be between say 2 (or less) and say 3-4. Much higher is a
>> >sign of a
>> >>>> problem.
>> >>>>
>> >>>>
>> >>>>> Attached is the ksp monitor/view output for gamg on the
>> >unsuccessful
>> >>>>> solve
>> >>>>>
>> >>>>> I'm also attaching a zip file which contains the simple PETSc
>> >script
>> >>>>> that loads the binary matrix/vector as well as two test cases, if
>> >you guys
>> >>>>> are interested in trying it out. It only works if you have PETSc
>> >configured
>> >>>>> with complex numbers.
>> >>>>>
>> >>>>> Thanks
>> >>>>>
>> >>>>> Justin
>> >>>>>
>> >>>>> PS - A couple years ago I had asked if there was a paper/tutorial
>> >on
>> >>>>> using/tuning GAMG. Does such a thing exist today?
>> >>>>>
>> >>>>
>> >>>> There is a write up in the manual that is tutorial like.
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
>> ><knepley at gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
>> ><jychang48 at gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
>> >to
>> >>>>>>> solve:
>> >>>>>>>
>> >>>>>>>
>> >http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>> >>>>>>>
>> >>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>> >inverting
>> >>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
>> >equations
>> >>>>>>> like those in the next section.
>> >>>>>>>
>> >>>>>>
>> >>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>> >>>>>> M-matrix to me (if all the y's are positive). This means
>> >>>>>> that it should be really easy to solve, and I think GAMG should
>> >do it.
>> >>>>>> You can start out just doing relaxation, like SOR, on
>> >>>>>> small examples.
>> >>>>>>
>> >>>>>>   Thanks,
>> >>>>>>
>> >>>>>>     Matt
>> >>>>>>
>> >>>>>>
>> >>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
>> ><knepley at gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>> >>>>>>>> petsc-users at mcs.anl.gov> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi all,
>> >>>>>>>>>
>> >>>>>>>>> I'm working with some folks to extract a linear system of
>> >equations
>> >>>>>>>>> from an external software package that solves power flow
>> >equations in
>> >>>>>>>>> complex form. Since that external package uses serial direct
>> >solvers like
>> >>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same
>> >matrix can
>> >>>>>>>>> be solved in PETSc using its parallel solvers.
>> >>>>>>>>>
>> >>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>> >>>>>>>>> processes on a single node (went from solving a 300k dog
>> >system in 1.8
>> >>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
>> >solvers and
>> >>>>>>>>> preconditioners but I have never worked with complex numbers
>> >so I am not
>> >>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>> >>>>>>>>>
>> >>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
>> >(unsurprisingly). I
>> >>>>>>>>> believe I also tried BICG with BJACOBI and while it did
>> >converge it
>> >>>>>>>>> converged slowly. Does anyone have recommendations on how one
>> >would go
>> >>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
>> >was originally
>> >>>>>>>>> thinking about converting it to cartesian form: Declaring all
>> >voltages =
>> >>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a
>> >conditional
>> >>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
>> >literature that
>> >>>>>>>>> claim to successfully precondition power flow equations
>> >operate in this
>> >>>>>>>>> form.
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 1) We really need to see the (simplified) equations
>> >>>>>>>>
>> >>>>>>>> 2) All complex equations can be converted to a system of real
>> >>>>>>>> equations twice as large, but this is not necessarily the best
>> >way to go
>> >>>>>>>>
>> >>>>>>>>  Thanks,
>> >>>>>>>>
>> >>>>>>>>     Matt
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> Justin
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> What most experimenters take for granted before they begin
>> >their
>> >>>>>>>> experiments is infinitely more interesting than any results to
>> >which their
>> >>>>>>>> experiments lead.
>> >>>>>>>> -- Norbert Wiener
>> >>>>>>>>
>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/
>> >>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> What most experimenters take for granted before they begin their
>> >>>>>> experiments is infinitely more interesting than any results to
>> >which their
>> >>>>>> experiments lead.
>> >>>>>> -- Norbert Wiener
>> >>>>>>
>> >>>>>> https://www.cse.buffalo.edu/~knepley/
>> >>>>>> <http://www.cse.buffalo.edu/~knepley/>
>> >>>>>>
>> >>>>>
>> >>
>> >> --
>> >> What most experimenters take for granted before they begin their
>> >> experiments is infinitely more interesting than any results to which
>> >their
>> >> experiments lead.
>> >> -- Norbert Wiener
>> >>
>> >> https://www.cse.buffalo.edu/~knepley/
>> >> <http://www.cse.buffalo.edu/~knepley/>
>> >>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/5d6408fc/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri Feb  1 16:34:08 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 1 Feb 2019 22:34:08 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
Message-ID: <0D0061EA-E1C5-4EF7-B351-2FC8C75F7819@anl.gov>


  Some of us have found that overlapping additive Schwarz can (at least sometimes) work well (has good scalability and good solution time) for this kind of problem. You can try -pc_type asm 

  Also look at changing the overlap from the default -pc_asm_overlap 2

  Also maybe try direct solvers on the subdomains -sub_pc_type lu

  If you have large problems on each process you might consider using more subdomains than processors with -pc_asm_local_blocks 

  Of course if GAMG is able to get good convergence rates consistently than it is fine to stick with that.

  Barry



  

> On Feb 1, 2019, at 1:38 AM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> So my matrix doesn't quite look like that, since I am solving a three-phase unbalanced system. Here's an example of what my matrix looks like for the IEEE-4 bus system with three-phases (so 12 total dofs): 
> 
> Mat Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 344.553 - 1240.43 i) (1, 30.7034 + 10.9125 i) (2, 31.1021 + 10.5924 i) (3, -1.28446 + 2.66353 i) (4, 0.623954 - 0.914098 i) (5, 0.225182 - 0.593995 i)
> row 1: (0, 30.7034 + 10.9125 i) (1, 344.705 - 1240.53 i) (2, 30.945 + 10.7235 i) (3, 0.623954 - 0.914098 i) (4, -1.43666 + 2.76012 i) (5, 0.382357 - 0.725045 i)
> row 2: (0, 31.1021 + 10.5924 i) (1, 30.945 + 10.7235 i) (2, 344.423 - 1240.34 i) (3, 0.225182 - 0.593995 i) (4, 0.382357 - 0.725045 i) (5, -1.15424 + 2.57087 i)
> row 3: (0, -1.28446 + 2.66353 i) (1, 0.623954 - 0.914098 i) (2, 0.225182 - 0.593995 i) (3, 1.38874 - 3.28923 i) (4, -0.623954 + 0.914098 i) (5, -0.225182 + 0.593995 i) (6, -0.312601 + 1.8756 i)
> row 4: (0, 0.623954 - 0.914098 i) (1, -1.43666 + 2.76012 i) (2, 0.382357 - 0.725045 i) (3, -0.623954 + 0.914098 i) (4, 1.54094 - 3.38582 i) (5, -0.382357 + 0.725045 i) (7, -0.312601 + 1.8756 i)
> row 5: (0, 0.225182 - 0.593995 i) (1, 0.382357 - 0.725045 i) (2, -1.15424 + 2.57087 i) (3, -0.225182 + 0.593995 i) (4, -0.382357 + 0.725045 i) (5, 1.25852 - 3.19657 i) (8, -0.312601 + 1.8756 i)
> row 6: (3, -0.312601 + 1.8756 i) (6, 1.96462 - 7.75312 i) (7, -0.499163 + 0.731278 i) (8, -0.180145 + 0.475196 i) (9, -1.02757 + 2.13082 i) (10, 0.499163 - 0.731279 i) (11, 0.180145 - 0.475196 i)
> row 7: (4, -0.312601 + 1.8756 i) (6, -0.499163 + 0.731278 i) (7, 2.08637 - 7.8304 i) (8, -0.305885 + 0.580036 i) (9, 0.499163 - 0.731279 i) (10, -1.14932 + 2.2081 i) (11, 0.305885 - 0.580036 i)
> row 8: (5, -0.312601 + 1.8756 i) (6, -0.180145 + 0.475196 i) (7, -0.305885 + 0.580036 i) (8, 1.86044 - 7.679 i) (9, 0.180145 - 0.475196 i) (10, 0.305885 - 0.580036 i) (11, -0.923391 + 2.0567 i)
> row 9: (6, -1.02757 + 2.13082 i) (7, 0.499163 - 0.731279 i) (8, 0.180145 - 0.475196 i) (9, 1.3396 - 2.28195 i) (10, -0.499163 + 0.731278 i) (11, -0.180145 + 0.475196 i)
> row 10: (6, 0.499163 - 0.731279 i) (7, -1.14932 + 2.2081 i) (8, 0.305885 - 0.580036 i) (9, -0.499163 + 0.731278 i) (10, 1.46136 - 2.35922 i) (11, -0.305885 + 0.580036 i)
> row 11: (6, 0.180145 - 0.475196 i) (7, 0.305885 - 0.580036 i) (8, -0.923391 + 2.0567 i) (9, -0.180145 + 0.475196 i) (10, -0.305885 + 0.580036 i) (11, 1.23543 - 2.20782 i)
> 
> Both GAMG and ILU are nice and dandy for this, but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG. Attached is the ksp monitor/view output for gamg on the unsuccessful solve
> 
> I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
> 
> Thanks
> 
> Justin
> 
> PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
> 
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:
> Here's IMHO the simplest explanation of the equations I'm trying to solve:
> 
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> 
> Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
> 
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> small examples.
> 
>   Thanks,
> 
>     Matt
>  
> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi all,
> 
> I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers. 
> 
> I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
> 
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
> 
> 1) We really need to see the (simplified) equations
> 
> 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
> 
>  Thanks,
> 
>     Matt
>  
> Justin
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <pcgamg_ksp_monitor.txt><solveYmat.zip>


From bsmith at mcs.anl.gov  Fri Feb  1 16:50:09 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 1 Feb 2019 22:50:09 +0000
Subject: [petsc-users] petsc4py for numpy array <-> MATDENSE binary
In-Reply-To: <CAOGsD9gWWTCGAJVMgBs9rgF16uAUY_XGqBqf02=xzXLz+yFsiQ@mail.gmail.com>
References: <CAOGsD9gWWTCGAJVMgBs9rgF16uAUY_XGqBqf02=xzXLz+yFsiQ@mail.gmail.com>
Message-ID: <AE364A90-0831-4E85-9783-6E2B6FF95648@anl.gov>


  Hmm, do you really want to store it as a PETSc dense matrix, or rather as a PETSc vector? I ask because it is natural to think of multidimensional arrays unrolled as a vector but a little strange for a multidimensional array to also be a dense matrix OPERATOR (since PETSc dense matrices are really linear operators that map vectors not 2 dimensional representations of vectors).

  Anyways in C you would use MatCreateDense() and pass the 3D array (conceptually unrolled into a one dimensional array) as input. I do not know if PETSc4py has similar functionality.

  Barry


> On Jan 31, 2019, at 7:07 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi, 
> 
> I have a large (~25k x 25k x 50) 3D array that I want to store as binary PETSc readable matrix using the MATDENSE format. I've currently achieved this (for smaller arrays) by writing the matrix out to an ASCII file and converting this ASCII file to binary, one number at a time using a C script. 
> 
> Does petsc4py support converting a 3d numpy array (complex valued) to MATDENSE (MPI) binary format directly ? 
> 
> I saw some discussion on this topic back in 2012 on the mailing list but it's not clear to me what happened after that. 
> 
> Thank You, 
> Sajid Ali
> Applied Physics
> Northwestern University


From sajidsyed2021 at u.northwestern.edu  Fri Feb  1 17:12:35 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Fri, 1 Feb 2019 17:12:35 -0600
Subject: [petsc-users] petsc4py for numpy array <-> MATDENSE binary
In-Reply-To: <AE364A90-0831-4E85-9783-6E2B6FF95648@anl.gov>
References: <CAOGsD9gWWTCGAJVMgBs9rgF16uAUY_XGqBqf02=xzXLz+yFsiQ@mail.gmail.com>
	<AE364A90-0831-4E85-9783-6E2B6FF95648@anl.gov>
Message-ID: <CAOGsD9gN=J6fhp=mAmAYmupRPLD1ykritEnRMyHPxk9mkCQ_CQ@mail.gmail.com>

The vector is essentially snapshots in time of a data array. I should
probably store this as a 2D dense matrix of dimensions (dim_x*dim_y) *
dim_z. Now I can pick one column at a time and use it for my TS Jacobian.
Apologies for being a little unclear.

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/0b0404d3/attachment.html>

From mfadams at lbl.gov  Fri Feb  1 21:04:56 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 1 Feb 2019 22:04:56 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAJ98EDqeogf3pLb00KLw5z-CL03SKiX8U-SCZZdp992NQr9LSA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
	<CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>
	<CAJ98EDqeogf3pLb00KLw5z-CL03SKiX8U-SCZZdp992NQr9LSA@mail.gmail.com>
Message-ID: <CADOhEh5Ghug2YB29uxUHBAdi0+vmS5XeqW1nvk-mFA5aa+-W8w@mail.gmail.com>

On Fri, Feb 1, 2019 at 5:32 PM Dave May <dave.mayhem23 at gmail.com> wrote:

> I'd suggest
> 1) using LU on the coarse grid (you have 1 application of SOR) and
>

Ah, good catch!

By setting every level KSP/PC explicitly you seemed to have overroad the
coarse grid LU solver that is set by default in GAMG.


> 2) do more Richardson iterations on each level (you have the default of
> 2). Try 4 or 6
>
> Thanks,
>   Dave
>
> On Fri, 1 Feb 2019 at 22:18, Mark Adams via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> We do need the equations and discretization, this not a Laplacian or
>> something is not working out-of-th-box.
>>
>> For pc_gamg_agg_nsmooths 0 you want  -pc_gamg_square_graph 10
>>
>> The info output snows the size of the systems on each level and the
>> number of non-zeros. with pc_gamg_agg_nsmooths 0 in 2D it courses very
>> slowly. You want a reduction factor between grids of say 4 - 10 between
>> grids.
>>
>> And here is shorthand for r/sor on all levels:
>>
>> -mg_levels_ksp_type richardson
>> -mg_levels_pc_type sor
>>
>> On Fri, Feb 1, 2019 at 4:07 PM Isaac, Tobin G via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> - Resuggesting a look at the max/min eigenvalues for as big a system as
>>> possible.
>>> - I like doing fgmres with a monitored krylov method to see if the
>>> smoother is behaving very differently on the fine vs intermediate vs coarse
>>> levels
>>>
>>>
>>> On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >I tried these options:
>>> >
>>> >-ksp_view
>>> >-ksp_monitor_true_residual
>>> >-ksp_type gmres
>>> >-ksp_max_it 50
>>> >-pc_type gamg
>>> >-mg_coarse_pc_type sor
>>> >-mg_levels_1_ksp_type richardson
>>> >-mg_levels_2_ksp_type richardson
>>> >-mg_levels_3_ksp_type richardson
>>> >-mg_levels_4_ksp_type richardson
>>> >-mg_levels_1_pc_type sor
>>> >-mg_levels_2_pc_type sor
>>> >-mg_levels_3_pc_type sor
>>> >-mg_levels_4_pc_type sor
>>> >
>>> >And still have a non-converging solution:
>>> >
>>> >  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>>> >1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>>> >  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>>> >1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>>> >  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>>> >1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>>> >  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>>> >1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>>> >  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>>> >1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>>> >  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>>> >1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>>> >  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>>> >1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>>> >  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>>> >1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>>> >  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>>> >1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>>> >  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>>> >1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>>> > 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>>> >1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>>> > 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>>> >1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>>> > 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>>> >1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>>> > 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>>> >1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>>> > 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>>> >1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>>> > 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>>> >1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>>> > 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>>> >1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>>> > 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>>> >1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>>> > 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>>> >1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>>> > 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>>> >1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>>> > 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>>> >1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>>> > 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>>> >1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>>> > 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>>> >1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>>> > 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>>> >1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>>> > 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>>> >1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>>> > 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>>> >1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>>> > 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>>> >1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>>> > 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>>> >1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>>> > 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>>> >1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>>> > 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>>> >1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>>> > 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>>> >1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>>> > 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>>> >1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>>> > 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>>> >1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>>> > 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>>> >1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>>> > 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>>> >1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>>> > 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>>> >1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>>> > 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>>> >1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>>> > 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>>> >1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>>> > 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>>> >1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>>> > 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>>> >1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>>> > 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>>> >1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>>> > 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>>> >1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>>> > 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>>> >1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>>> > 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>>> >1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>>> > 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>>> >1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>>> > 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>>> >1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>>> > 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>>> >1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>>> > 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>>> >1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>>> > 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>>> >1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>>> > 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>>> >1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>>> > 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>>> >1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>>> >KSP Object: 1 MPI processes
>>> >  type: gmres
>>> >restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
>>> >with no iterative refinement
>>> >    happy breakdown tolerance 1e-30
>>> >  maximum iterations=50, initial guess is zero
>>> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >  left preconditioning
>>> >  using PRECONDITIONED norm type for convergence test
>>> >PC Object: 1 MPI processes
>>> >  type: gamg
>>> >    type is MULTIPLICATIVE, levels=5 cycles=v
>>> >      Cycles per PCApply=1
>>> >      Using externally compute Galerkin coarse grid matrices
>>> >      GAMG specific options
>>> >      Threshold for dropping small values in graph on each level =   0.
>>> >0.   0.
>>> >        Threshold scaling factor for each level not specified = 1.
>>> >        AGG specific options
>>> >          Symmetric graph false
>>> >          Number of levels to square graph 1
>>> >          Number smoothing steps 1
>>> >        Complexity:    grid = 1.31821
>>> >  Coarse grid solver -- level -------------------------------
>>> >    KSP Object: (mg_coarse_) 1 MPI processes
>>> >      type: preonly
>>> >      maximum iterations=10000, initial guess is zero
>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >      left preconditioning
>>> >      using NONE norm type for convergence test
>>> >    PC Object: (mg_coarse_) 1 MPI processes
>>> >      type: sor
>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>> >= 1.
>>> >      linear system matrix = precond matrix:
>>> >      Mat Object: 1 MPI processes
>>> >        type: seqaij
>>> >        rows=17, cols=17
>>> >        total: nonzeros=163, allocated nonzeros=163
>>> >        total number of mallocs used during MatSetValues calls =0
>>> >          using I-node routines: found 12 nodes, limit used is 5
>>> >  Down solver (pre-smoother) on level 1 -------------------------------
>>> >    KSP Object: (mg_levels_1_) 1 MPI processes
>>> >      type: richardson
>>> >        damping factor=1.
>>> >      maximum iterations=2, nonzero initial guess
>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >      left preconditioning
>>> >      using NONE norm type for convergence test
>>> >    PC Object: (mg_levels_1_) 1 MPI processes
>>> >      type: sor
>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>> >= 1.
>>> >      linear system matrix = precond matrix:
>>> >      Mat Object: 1 MPI processes
>>> >        type: seqaij
>>> >        rows=100, cols=100
>>> >        total: nonzeros=1240, allocated nonzeros=1240
>>> >        total number of mallocs used during MatSetValues calls =0
>>> >          not using I-node routines
>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>> >  Down solver (pre-smoother) on level 2 -------------------------------
>>> >    KSP Object: (mg_levels_2_) 1 MPI processes
>>> >      type: richardson
>>> >        damping factor=1.
>>> >      maximum iterations=2, nonzero initial guess
>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >      left preconditioning
>>> >      using NONE norm type for convergence test
>>> >    PC Object: (mg_levels_2_) 1 MPI processes
>>> >      type: sor
>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>> >= 1.
>>> >      linear system matrix = precond matrix:
>>> >      Mat Object: 1 MPI processes
>>> >        type: seqaij
>>> >        rows=537, cols=537
>>> >        total: nonzeros=5291, allocated nonzeros=5291
>>> >        total number of mallocs used during MatSetValues calls =0
>>> >          not using I-node routines
>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>> >  Down solver (pre-smoother) on level 3 -------------------------------
>>> >    KSP Object: (mg_levels_3_) 1 MPI processes
>>> >      type: richardson
>>> >        damping factor=1.
>>> >      maximum iterations=2, nonzero initial guess
>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >      left preconditioning
>>> >      using NONE norm type for convergence test
>>> >    PC Object: (mg_levels_3_) 1 MPI processes
>>> >      type: sor
>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>> >= 1.
>>> >      linear system matrix = precond matrix:
>>> >      Mat Object: 1 MPI processes
>>> >        type: seqaij
>>> >        rows=1541, cols=1541
>>> >        total: nonzeros=8039, allocated nonzeros=8039
>>> >        total number of mallocs used during MatSetValues calls =0
>>> >          not using I-node routines
>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>> >  Down solver (pre-smoother) on level 4 -------------------------------
>>> >    KSP Object: (mg_levels_4_) 1 MPI processes
>>> >      type: richardson
>>> >        damping factor=1.
>>> >      maximum iterations=2, nonzero initial guess
>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>> >      left preconditioning
>>> >      using NONE norm type for convergence test
>>> >    PC Object: (mg_levels_4_) 1 MPI processes
>>> >      type: sor
>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>> >= 1.
>>> >      linear system matrix = precond matrix:
>>> >      Mat Object: 1 MPI processes
>>> >        type: seqaij
>>> >        rows=8541, cols=8541
>>> >        total: nonzeros=46299, allocated nonzeros=46299
>>> >        total number of mallocs used during MatSetValues calls =0
>>> >          using I-node routines: found 5464 nodes, limit used is 5
>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>> >  linear system matrix = precond matrix:
>>> >  Mat Object: 1 MPI processes
>>> >    type: seqaij
>>> >    rows=8541, cols=8541
>>> >    total: nonzeros=46299, allocated nonzeros=46299
>>> >    total number of mallocs used during MatSetValues calls =0
>>> >      using I-node routines: found 5464 nodes, limit used is 5
>>> >
>>> >Am I doing this right? Did I miss anything?
>>> >
>>> >
>>> >
>>> >On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>>> >wrote:
>>> >
>>> >> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>>> >wrote:
>>> >>
>>> >>> Hi Mark,
>>> >>>
>>> >>> 1) So with these options:
>>> >>>
>>> >>> -ksp_type gmres
>>> >>> -ksp_rtol 1e-15
>>> >>> -ksp_monitor_true_residual
>>> >>> -ksp_converged_reason
>>> >>> -pc_type bjacobi
>>> >>>
>>> >>> This is what I get:
>>> >>>
>>> >>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>> >>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>> >>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>> >>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>> >>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>> >>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>> >>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>> >>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>> >>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>> >>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>> >>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>> >>>
>>> >>> 2) With richardson/sor:
>>> >>>
>>> >>
>>> >> Okay, its looks like Richardson/SOR solves this just fine. You can
>>> >use
>>> >> this as the smoother for GAMG instead
>>> >> of Cheby/Jacobi, and probably see better results on the larger
>>> >problems.
>>> >>
>>> >>   Matt
>>> >>
>>> >>
>>> >>> -ksp_type richardson
>>> >>> -ksp_rtol 1e-15
>>> >>> -ksp_monitor_true_residual
>>> >>> -pc_type sor
>>> >>>
>>> >>> This is what I get:
>>> >>>
>>> >>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>> >>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>> >>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>> >>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>> >>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>> >>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>> >>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>> >>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>> >>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>> >>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>> >>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>> >>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>> >>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>> >>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>> >>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>> >>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>> >>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>> >>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>> >>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>> >>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>> >>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>> >>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>> >>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>> >>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>> >>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>> >>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>> >>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>> >>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>> >>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>> >>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>> >>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>> >>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>> >>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>> >>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>> >>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>> >>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>> >>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>> >>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>> >>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>> >>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>> >>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>> >>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>> >>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>> >>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>> >>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>> >>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>> >>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>> >>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>> >>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>> >>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>> >>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>> >>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>> >>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>> >>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>> >>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>> >>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>> >>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>> >>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>> >>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>> >>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>> >>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>> >>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>> >>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>> >>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>> >>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>> >>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>> >>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>> >>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>> >>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>> >>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>> >>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>> >>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>> >>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>> >>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>> >>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>> >>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>> >>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>> >>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>> >>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>> >>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>> >>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>> >>>
>>> >>> 3) And lastly with chebyshev/jacobi:
>>> >>>
>>> >>> -ksp_type chebyshev
>>> >>> -ksp_rtol 1e-15
>>> >>> -ksp_monitor_true_residual
>>> >>> -pc_type jacobi
>>> >>>
>>> >>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>> >>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>> >>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>> >>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>> >>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>> >>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>> >>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>> >>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>> >>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>> >>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>> >>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>> >>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>> >>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>> >>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>> >>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>> >>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>> >>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>> >>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>> >>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>> >>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>> >>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>> >>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>> >>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>> >>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>> >>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>> >>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>> >>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>> >>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>> >>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>> >>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>> >>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>> >>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>> >>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>> >>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>> >>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>> >>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>> >>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>> >>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>> >>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>> >>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>> >>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>> >>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>> >>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>> >>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>> >>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>> >>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>> >>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>> >>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>> >>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>> >>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>> >>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>> >>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>> >>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>> >>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>> >>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>> >>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>> >>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>> >>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>> >>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>> >>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>> >>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>> >>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>> >>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>> >>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>> >>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>> >>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>> >>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>> >>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>> >>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>> >>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>> >>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>> >>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>> >>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>> >>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>> >>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>> >>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>> >>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>> >>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>> >>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>> >>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>> >>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>> >>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>> >>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>> >>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>> >>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>> >>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>> >>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>> >>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>> >>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>> >>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>> >>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>> >>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>> >>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>> >>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>> >>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>> >>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>> >>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>> >>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>> >>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>> >>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>> >>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>> >>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>> >>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>> >>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>> >>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>> >>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>> >>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>> >>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>> >>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>> >>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>> >>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>> >>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>> >>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>> >>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>> >>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>> >>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>> >>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>> >>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>> >>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>> >>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>> >>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>> >>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>> >>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>> >>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>> >>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>> >>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>> >>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>> >>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>> >>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>> >>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>> >>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>> >>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>> >>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>> >>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>> >>>
>>> >>> Looks like neither of these ksp/pc combos are good? I also tried
>>> >>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
>>> >Here's the
>>> >>> GAMG info I was able to grep from the out-of-box params:
>>> >>>
>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> >>> nnz/row (ave)=5, np=1
>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>> >threshold
>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>> >>> min=2.394056e-02 PC=jacobi
>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>>> >active
>>> >>> pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 5.21674 nnz ave. (N=1541)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>> >>> min=6.783380e-02 PC=jacobi
>>> >>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 9.85289 nnz ave. (N=537)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>> >>> min=5.974776e-02 PC=jacobi
>>> >>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>>> >active
>>> >>> pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 12.4 nnz ave. (N=100)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>> >>> min=4.842076e-01 PC=jacobi
>>> >>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
>>> >active pes
>>> >>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>> >>>
>>> >>> And the one with pc_gamg_agg_nsmooths 0
>>> >>>
>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> >>> nnz/row (ave)=5, np=1
>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>> >threshold
>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>>> >active
>>> >>> pes
>>> >>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
>>> >threshold
>>> >>> 0., 3.07398 nnz ave. (N=1541)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>> >>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 3.02211 nnz ave. (N=814)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>> >>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 3.00434 nnz ave. (N=461)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>> >>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 3. nnz ave. (N=290)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>> >>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 3. nnz ave. (N=197)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>> >>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 2.98425 nnz ave. (N=127)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>> >>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 2.97561 nnz ave. (N=82)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>> >>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
>>> >active pes
>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>> >0.,
>>> >>> 2.9697 nnz ave. (N=66)
>>> >>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>> >>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
>>> >active pes
>>> >>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> Justin
>>> >>>
>>> >>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>> >>>
>>> >>>>
>>> >>>>
>>> >>>>> Both GAMG and ILU are nice and dandy for this,
>>> >>>>>
>>> >>>>
>>> >>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>>> >and
>>> >>>> converge it way down, say rtol = 1.e-12. See which one is better in
>>> >the
>>> >>>> early iteration and pick it. It would be nice to check that it
>>> >solves the
>>> >>>> problem ...
>>> >>>>
>>> >>>> The residual drops about 5 orders in the first iteration and then
>>> >>>> flatlines. That is very bad. Check that the smoother can actually
>>> >solve the
>>> >>>> problem.
>>> >>>>
>>> >>>>
>>> >>>>> but as soon as I look at a bigger system, like a network with 8500
>>> >>>>> buses, the out-of-box gamg craps out. I am not sure where to start
>>> >when it
>>> >>>>> comes to tuning GAMG.
>>> >>>>>
>>> >>>>
>>> >>>> First try -pc_gamg_nsmooths 0
>>> >>>>
>>> >>>> You can run with -info and grep on GAMG to see some diagnostic
>>> >output.
>>> >>>> Eigen estimates are fragile in practice and with this parameter and
>>> >SOR
>>> >>>> there are no eigen estimates needed. The max eigen values for all
>>> >levels
>>> >>>> should be between say 2 (or less) and say 3-4. Much higher is a
>>> >sign of a
>>> >>>> problem.
>>> >>>>
>>> >>>>
>>> >>>>> Attached is the ksp monitor/view output for gamg on the
>>> >unsuccessful
>>> >>>>> solve
>>> >>>>>
>>> >>>>> I'm also attaching a zip file which contains the simple PETSc
>>> >script
>>> >>>>> that loads the binary matrix/vector as well as two test cases, if
>>> >you guys
>>> >>>>> are interested in trying it out. It only works if you have PETSc
>>> >configured
>>> >>>>> with complex numbers.
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>>
>>> >>>>> Justin
>>> >>>>>
>>> >>>>> PS - A couple years ago I had asked if there was a paper/tutorial
>>> >on
>>> >>>>> using/tuning GAMG. Does such a thing exist today?
>>> >>>>>
>>> >>>>
>>> >>>> There is a write up in the manual that is tutorial like.
>>> >>>>
>>> >>>>
>>> >>>>>
>>> >>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
>>> ><knepley at gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
>>> ><jychang48 at gmail.com>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
>>> >to
>>> >>>>>>> solve:
>>> >>>>>>>
>>> >>>>>>>
>>> >http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>> >>>>>>>
>>> >>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>>> >inverting
>>> >>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
>>> >equations
>>> >>>>>>> like those in the next section.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>> >>>>>> M-matrix to me (if all the y's are positive). This means
>>> >>>>>> that it should be really easy to solve, and I think GAMG should
>>> >do it.
>>> >>>>>> You can start out just doing relaxation, like SOR, on
>>> >>>>>> small examples.
>>> >>>>>>
>>> >>>>>>   Thanks,
>>> >>>>>>
>>> >>>>>>     Matt
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
>>> ><knepley at gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>> >>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Hi all,
>>> >>>>>>>>>
>>> >>>>>>>>> I'm working with some folks to extract a linear system of
>>> >equations
>>> >>>>>>>>> from an external software package that solves power flow
>>> >equations in
>>> >>>>>>>>> complex form. Since that external package uses serial direct
>>> >solvers like
>>> >>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same
>>> >matrix can
>>> >>>>>>>>> be solved in PETSc using its parallel solvers.
>>> >>>>>>>>>
>>> >>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>> >>>>>>>>> processes on a single node (went from solving a 300k dog
>>> >system in 1.8
>>> >>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
>>> >solvers and
>>> >>>>>>>>> preconditioners but I have never worked with complex numbers
>>> >so I am not
>>> >>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>> >>>>>>>>>
>>> >>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
>>> >(unsurprisingly). I
>>> >>>>>>>>> believe I also tried BICG with BJACOBI and while it did
>>> >converge it
>>> >>>>>>>>> converged slowly. Does anyone have recommendations on how one
>>> >would go
>>> >>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
>>> >was originally
>>> >>>>>>>>> thinking about converting it to cartesian form: Declaring all
>>> >voltages =
>>> >>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a
>>> >conditional
>>> >>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
>>> >literature that
>>> >>>>>>>>> claim to successfully precondition power flow equations
>>> >operate in this
>>> >>>>>>>>> form.
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> 1) We really need to see the (simplified) equations
>>> >>>>>>>>
>>> >>>>>>>> 2) All complex equations can be converted to a system of real
>>> >>>>>>>> equations twice as large, but this is not necessarily the best
>>> >way to go
>>> >>>>>>>>
>>> >>>>>>>>  Thanks,
>>> >>>>>>>>
>>> >>>>>>>>     Matt
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>> Justin
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> --
>>> >>>>>>>> What most experimenters take for granted before they begin
>>> >their
>>> >>>>>>>> experiments is infinitely more interesting than any results to
>>> >which their
>>> >>>>>>>> experiments lead.
>>> >>>>>>>> -- Norbert Wiener
>>> >>>>>>>>
>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>> >>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> What most experimenters take for granted before they begin their
>>> >>>>>> experiments is infinitely more interesting than any results to
>>> >which their
>>> >>>>>> experiments lead.
>>> >>>>>> -- Norbert Wiener
>>> >>>>>>
>>> >>>>>> https://www.cse.buffalo.edu/~knepley/
>>> >>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>> >>>>>>
>>> >>>>>
>>> >>
>>> >> --
>>> >> What most experimenters take for granted before they begin their
>>> >> experiments is infinitely more interesting than any results to which
>>> >their
>>> >> experiments lead.
>>> >> -- Norbert Wiener
>>> >>
>>> >> https://www.cse.buffalo.edu/~knepley/
>>> >> <http://www.cse.buffalo.edu/~knepley/>
>>> >>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190201/35410dd7/attachment-0001.html>

From dave.mayhem23 at gmail.com  Sat Feb  2 04:44:40 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sat, 2 Feb 2019 11:44:40 +0100
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CADOhEh5Ghug2YB29uxUHBAdi0+vmS5XeqW1nvk-mFA5aa+-W8w@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
	<CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>
	<CAJ98EDqeogf3pLb00KLw5z-CL03SKiX8U-SCZZdp992NQr9LSA@mail.gmail.com>
	<CADOhEh5Ghug2YB29uxUHBAdi0+vmS5XeqW1nvk-mFA5aa+-W8w@mail.gmail.com>
Message-ID: <CAJ98EDrTL34Q3z_+gpqz95nQ-7Yx40QeOWfS08TYJtxdEPm84Q@mail.gmail.com>

On Sat, 2 Feb 2019 at 04:05, Mark Adams <mfadams at lbl.gov> wrote:

>
>
> On Fri, Feb 1, 2019 at 5:32 PM Dave May <dave.mayhem23 at gmail.com> wrote:
>
>> I'd suggest
>> 1) using LU on the coarse grid (you have 1 application of SOR) and
>>
>
> Ah, good catch!
>
> By setting every level KSP/PC explicitly you seemed to have overroad the
> coarse grid LU solver that is set by default in GAMG.
>

Not really - there was an explicit option provided specifying the coarse pc
to use SOR.



>
>> 2) do more Richardson iterations on each level (you have the default of
>> 2). Try 4 or 6
>>
>> Thanks,
>>   Dave
>>
>> On Fri, 1 Feb 2019 at 22:18, Mark Adams via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> We do need the equations and discretization, this not a Laplacian or
>>> something is not working out-of-th-box.
>>>
>>> For pc_gamg_agg_nsmooths 0 you want  -pc_gamg_square_graph 10
>>>
>>> The info output snows the size of the systems on each level and the
>>> number of non-zeros. with pc_gamg_agg_nsmooths 0 in 2D it courses very
>>> slowly. You want a reduction factor between grids of say 4 - 10 between
>>> grids.
>>>
>>> And here is shorthand for r/sor on all levels:
>>>
>>> -mg_levels_ksp_type richardson
>>> -mg_levels_pc_type sor
>>>
>>> On Fri, Feb 1, 2019 at 4:07 PM Isaac, Tobin G via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> - Resuggesting a look at the max/min eigenvalues for as big a system as
>>>> possible.
>>>> - I like doing fgmres with a monitored krylov method to see if the
>>>> smoother is behaving very differently on the fine vs intermediate vs coarse
>>>> levels
>>>>
>>>>
>>>> On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> >I tried these options:
>>>> >
>>>> >-ksp_view
>>>> >-ksp_monitor_true_residual
>>>> >-ksp_type gmres
>>>> >-ksp_max_it 50
>>>> >-pc_type gamg
>>>> >-mg_coarse_pc_type sor
>>>> >-mg_levels_1_ksp_type richardson
>>>> >-mg_levels_2_ksp_type richardson
>>>> >-mg_levels_3_ksp_type richardson
>>>> >-mg_levels_4_ksp_type richardson
>>>> >-mg_levels_1_pc_type sor
>>>> >-mg_levels_2_pc_type sor
>>>> >-mg_levels_3_pc_type sor
>>>> >-mg_levels_4_pc_type sor
>>>> >
>>>> >And still have a non-converging solution:
>>>> >
>>>> >  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>>>> >1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>>>> >  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>>>> >1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>>>> >  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>>>> >1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>>>> >  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>>>> >1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>>>> >  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>>>> >1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>>>> >  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>>>> >1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>>>> >  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>>>> >1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>>>> >  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>>>> >1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>>>> >  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>>>> >1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>>>> >  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>>>> >1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>>>> > 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>>>> >1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>>>> > 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>>>> >1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>>>> > 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>>>> >1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>>>> > 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>>>> >1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>>>> > 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>>>> >1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>>>> > 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>>>> >1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>>>> > 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>>>> >1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>>>> > 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>>>> >1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>>>> > 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>>>> >1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>>>> > 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>>>> >1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>>>> > 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>>>> >1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>>>> > 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>>>> >1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>>>> > 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>>>> >1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>>>> > 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>>>> >1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>>>> > 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>>>> >1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>>>> > 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>>>> >1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>>>> > 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>>>> >1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>>>> > 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>>>> >1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>>>> > 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>>>> >1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>>>> > 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>>>> >1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>>>> > 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>>>> >1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>>>> > 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>>>> >1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>>>> > 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>>>> >1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>>>> > 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>>>> >1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>>>> > 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>>>> >1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>>>> > 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>>>> >1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>>>> > 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>>>> >1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>>>> > 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>>>> >1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>>>> > 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>>>> >1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>>>> > 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>>>> >1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>>>> > 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>>>> >1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>>>> > 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>>>> >1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>>>> > 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>>>> >1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>>>> > 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>>>> >1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>>>> > 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>>>> >1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>>>> > 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>>>> >1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>>>> > 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>>>> >1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>>>> > 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>>>> >1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>>>> > 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>>>> >1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>>>> > 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>>>> >1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>>>> > 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>>>> >1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>>>> >KSP Object: 1 MPI processes
>>>> >  type: gmres
>>>> >restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
>>>> >with no iterative refinement
>>>> >    happy breakdown tolerance 1e-30
>>>> >  maximum iterations=50, initial guess is zero
>>>> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >  left preconditioning
>>>> >  using PRECONDITIONED norm type for convergence test
>>>> >PC Object: 1 MPI processes
>>>> >  type: gamg
>>>> >    type is MULTIPLICATIVE, levels=5 cycles=v
>>>> >      Cycles per PCApply=1
>>>> >      Using externally compute Galerkin coarse grid matrices
>>>> >      GAMG specific options
>>>> >      Threshold for dropping small values in graph on each level =   0.
>>>> >0.   0.
>>>> >        Threshold scaling factor for each level not specified = 1.
>>>> >        AGG specific options
>>>> >          Symmetric graph false
>>>> >          Number of levels to square graph 1
>>>> >          Number smoothing steps 1
>>>> >        Complexity:    grid = 1.31821
>>>> >  Coarse grid solver -- level -------------------------------
>>>> >    KSP Object: (mg_coarse_) 1 MPI processes
>>>> >      type: preonly
>>>> >      maximum iterations=10000, initial guess is zero
>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >      left preconditioning
>>>> >      using NONE norm type for convergence test
>>>> >    PC Object: (mg_coarse_) 1 MPI processes
>>>> >      type: sor
>>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>>> >= 1.
>>>> >      linear system matrix = precond matrix:
>>>> >      Mat Object: 1 MPI processes
>>>> >        type: seqaij
>>>> >        rows=17, cols=17
>>>> >        total: nonzeros=163, allocated nonzeros=163
>>>> >        total number of mallocs used during MatSetValues calls =0
>>>> >          using I-node routines: found 12 nodes, limit used is 5
>>>> >  Down solver (pre-smoother) on level 1 -------------------------------
>>>> >    KSP Object: (mg_levels_1_) 1 MPI processes
>>>> >      type: richardson
>>>> >        damping factor=1.
>>>> >      maximum iterations=2, nonzero initial guess
>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >      left preconditioning
>>>> >      using NONE norm type for convergence test
>>>> >    PC Object: (mg_levels_1_) 1 MPI processes
>>>> >      type: sor
>>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>>> >= 1.
>>>> >      linear system matrix = precond matrix:
>>>> >      Mat Object: 1 MPI processes
>>>> >        type: seqaij
>>>> >        rows=100, cols=100
>>>> >        total: nonzeros=1240, allocated nonzeros=1240
>>>> >        total number of mallocs used during MatSetValues calls =0
>>>> >          not using I-node routines
>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>> >  Down solver (pre-smoother) on level 2 -------------------------------
>>>> >    KSP Object: (mg_levels_2_) 1 MPI processes
>>>> >      type: richardson
>>>> >        damping factor=1.
>>>> >      maximum iterations=2, nonzero initial guess
>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >      left preconditioning
>>>> >      using NONE norm type for convergence test
>>>> >    PC Object: (mg_levels_2_) 1 MPI processes
>>>> >      type: sor
>>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>>> >= 1.
>>>> >      linear system matrix = precond matrix:
>>>> >      Mat Object: 1 MPI processes
>>>> >        type: seqaij
>>>> >        rows=537, cols=537
>>>> >        total: nonzeros=5291, allocated nonzeros=5291
>>>> >        total number of mallocs used during MatSetValues calls =0
>>>> >          not using I-node routines
>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>> >  Down solver (pre-smoother) on level 3 -------------------------------
>>>> >    KSP Object: (mg_levels_3_) 1 MPI processes
>>>> >      type: richardson
>>>> >        damping factor=1.
>>>> >      maximum iterations=2, nonzero initial guess
>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >      left preconditioning
>>>> >      using NONE norm type for convergence test
>>>> >    PC Object: (mg_levels_3_) 1 MPI processes
>>>> >      type: sor
>>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>>> >= 1.
>>>> >      linear system matrix = precond matrix:
>>>> >      Mat Object: 1 MPI processes
>>>> >        type: seqaij
>>>> >        rows=1541, cols=1541
>>>> >        total: nonzeros=8039, allocated nonzeros=8039
>>>> >        total number of mallocs used during MatSetValues calls =0
>>>> >          not using I-node routines
>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>> >  Down solver (pre-smoother) on level 4 -------------------------------
>>>> >    KSP Object: (mg_levels_4_) 1 MPI processes
>>>> >      type: richardson
>>>> >        damping factor=1.
>>>> >      maximum iterations=2, nonzero initial guess
>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>> >      left preconditioning
>>>> >      using NONE norm type for convergence test
>>>> >    PC Object: (mg_levels_4_) 1 MPI processes
>>>> >      type: sor
>>>> >    type = local_symmetric, iterations = 1, local iterations = 1, omega
>>>> >= 1.
>>>> >      linear system matrix = precond matrix:
>>>> >      Mat Object: 1 MPI processes
>>>> >        type: seqaij
>>>> >        rows=8541, cols=8541
>>>> >        total: nonzeros=46299, allocated nonzeros=46299
>>>> >        total number of mallocs used during MatSetValues calls =0
>>>> >          using I-node routines: found 5464 nodes, limit used is 5
>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>> >  linear system matrix = precond matrix:
>>>> >  Mat Object: 1 MPI processes
>>>> >    type: seqaij
>>>> >    rows=8541, cols=8541
>>>> >    total: nonzeros=46299, allocated nonzeros=46299
>>>> >    total number of mallocs used during MatSetValues calls =0
>>>> >      using I-node routines: found 5464 nodes, limit used is 5
>>>> >
>>>> >Am I doing this right? Did I miss anything?
>>>> >
>>>> >
>>>> >
>>>> >On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>>>> >wrote:
>>>> >
>>>> >> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>>>> >wrote:
>>>> >>
>>>> >>> Hi Mark,
>>>> >>>
>>>> >>> 1) So with these options:
>>>> >>>
>>>> >>> -ksp_type gmres
>>>> >>> -ksp_rtol 1e-15
>>>> >>> -ksp_monitor_true_residual
>>>> >>> -ksp_converged_reason
>>>> >>> -pc_type bjacobi
>>>> >>>
>>>> >>> This is what I get:
>>>> >>>
>>>> >>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>> >>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>>> >>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>> >>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>>> >>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>> >>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>>> >>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>> >>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>>> >>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>> >>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>>> >>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>> >>>
>>>> >>> 2) With richardson/sor:
>>>> >>>
>>>> >>
>>>> >> Okay, its looks like Richardson/SOR solves this just fine. You can
>>>> >use
>>>> >> this as the smoother for GAMG instead
>>>> >> of Cheby/Jacobi, and probably see better results on the larger
>>>> >problems.
>>>> >>
>>>> >>   Matt
>>>> >>
>>>> >>
>>>> >>> -ksp_type richardson
>>>> >>> -ksp_rtol 1e-15
>>>> >>> -ksp_monitor_true_residual
>>>> >>> -pc_type sor
>>>> >>>
>>>> >>> This is what I get:
>>>> >>>
>>>> >>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>> >>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>>> >>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>> >>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>>> >>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>> >>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>>> >>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>> >>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>>> >>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>> >>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>>> >>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>> >>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>>> >>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>> >>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>>> >>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>> >>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>>> >>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>> >>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>>> >>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>> >>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>>> >>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>> >>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>>> >>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>> >>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>>> >>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>> >>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>>> >>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>> >>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>>> >>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>> >>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>>> >>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>> >>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>>> >>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>> >>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>>> >>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>> >>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>>> >>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>> >>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>>> >>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>> >>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>>> >>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>> >>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>>> >>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>> >>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>>> >>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>> >>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>>> >>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>> >>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>>> >>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>> >>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>>> >>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>> >>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>>> >>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>> >>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>>> >>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>> >>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>>> >>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>> >>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>>> >>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>> >>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>>> >>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>> >>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>>> >>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>> >>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>>> >>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>> >>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>>> >>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>> >>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>>> >>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>> >>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>>> >>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>> >>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>>> >>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>> >>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>>> >>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>> >>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>>> >>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>> >>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>>> >>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>> >>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>>> >>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>> >>>
>>>> >>> 3) And lastly with chebyshev/jacobi:
>>>> >>>
>>>> >>> -ksp_type chebyshev
>>>> >>> -ksp_rtol 1e-15
>>>> >>> -ksp_monitor_true_residual
>>>> >>> -pc_type jacobi
>>>> >>>
>>>> >>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>>> >>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>> >>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>>> >>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>> >>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>>> >>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>> >>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>>> >>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>> >>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>>> >>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>> >>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>>> >>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>> >>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>>> >>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>> >>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>>> >>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>> >>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>>> >>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>> >>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>>> >>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>> >>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>>> >>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>> >>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>>> >>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>> >>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>>> >>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>> >>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>>> >>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>> >>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>>> >>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>> >>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>>> >>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>> >>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>>> >>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>> >>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>>> >>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>> >>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>>> >>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>> >>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>>> >>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>> >>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>>> >>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>> >>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>>> >>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>> >>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>>> >>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>> >>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>>> >>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>> >>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>>> >>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>> >>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>>> >>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>> >>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>>> >>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>> >>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>>> >>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>> >>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>>> >>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>> >>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>>> >>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>> >>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>>> >>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>> >>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>>> >>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>> >>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>>> >>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>> >>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>>> >>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>> >>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>>> >>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>> >>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>>> >>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>> >>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>>> >>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>> >>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>>> >>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>> >>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>>> >>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>> >>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>>> >>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>> >>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>>> >>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>> >>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>>> >>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>> >>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>>> >>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>> >>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>>> >>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>> >>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>>> >>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>> >>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>>> >>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>> >>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>>> >>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>> >>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>>> >>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>> >>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>>> >>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>> >>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>>> >>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>> >>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>>> >>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>> >>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>>> >>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>> >>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>>> >>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>> >>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>>> >>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>> >>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>>> >>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>> >>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>>> >>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>> >>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>>> >>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>> >>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>>> >>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>> >>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>>> >>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>> >>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>>> >>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>> >>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>>> >>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>> >>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>>> >>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>> >>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>>> >>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>> >>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>>> >>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>> >>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>>> >>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>> >>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>>> >>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>> >>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>>> >>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>> >>>
>>>> >>> Looks like neither of these ksp/pc combos are good? I also tried
>>>> >>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
>>>> >Here's the
>>>> >>> GAMG info I was able to grep from the out-of-box params:
>>>> >>>
>>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>> >>> nnz/row (ave)=5, np=1
>>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>>> >threshold
>>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>>> >>> min=2.394056e-02 PC=jacobi
>>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>>>> >active
>>>> >>> pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 5.21674 nnz ave. (N=1541)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>>> >>> min=6.783380e-02 PC=jacobi
>>>> >>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 9.85289 nnz ave. (N=537)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>>> >>> min=5.974776e-02 PC=jacobi
>>>> >>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>>>> >active
>>>> >>> pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 12.4 nnz ave. (N=100)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>>> >>> min=4.842076e-01 PC=jacobi
>>>> >>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
>>>> >active pes
>>>> >>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>> >>>
>>>> >>> And the one with pc_gamg_agg_nsmooths 0
>>>> >>>
>>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>> >>> nnz/row (ave)=5, np=1
>>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>>> >threshold
>>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>>>> >active
>>>> >>> pes
>>>> >>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
>>>> >threshold
>>>> >>> 0., 3.07398 nnz ave. (N=1541)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>>> >>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 3.02211 nnz ave. (N=814)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>>> >>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 3.00434 nnz ave. (N=461)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>>> >>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 3. nnz ave. (N=290)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>>> >>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 3. nnz ave. (N=197)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>>> >>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 2.98425 nnz ave. (N=127)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>>> >>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 2.97561 nnz ave. (N=82)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>>> >>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
>>>> >active pes
>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>> >0.,
>>>> >>> 2.9697 nnz ave. (N=66)
>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>>> >>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
>>>> >active pes
>>>> >>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>> >>>
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Justin
>>>> >>>
>>>> >>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>> >>>
>>>> >>>>
>>>> >>>>
>>>> >>>>> Both GAMG and ILU are nice and dandy for this,
>>>> >>>>>
>>>> >>>>
>>>> >>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>>>> >and
>>>> >>>> converge it way down, say rtol = 1.e-12. See which one is better in
>>>> >the
>>>> >>>> early iteration and pick it. It would be nice to check that it
>>>> >solves the
>>>> >>>> problem ...
>>>> >>>>
>>>> >>>> The residual drops about 5 orders in the first iteration and then
>>>> >>>> flatlines. That is very bad. Check that the smoother can actually
>>>> >solve the
>>>> >>>> problem.
>>>> >>>>
>>>> >>>>
>>>> >>>>> but as soon as I look at a bigger system, like a network with 8500
>>>> >>>>> buses, the out-of-box gamg craps out. I am not sure where to start
>>>> >when it
>>>> >>>>> comes to tuning GAMG.
>>>> >>>>>
>>>> >>>>
>>>> >>>> First try -pc_gamg_nsmooths 0
>>>> >>>>
>>>> >>>> You can run with -info and grep on GAMG to see some diagnostic
>>>> >output.
>>>> >>>> Eigen estimates are fragile in practice and with this parameter and
>>>> >SOR
>>>> >>>> there are no eigen estimates needed. The max eigen values for all
>>>> >levels
>>>> >>>> should be between say 2 (or less) and say 3-4. Much higher is a
>>>> >sign of a
>>>> >>>> problem.
>>>> >>>>
>>>> >>>>
>>>> >>>>> Attached is the ksp monitor/view output for gamg on the
>>>> >unsuccessful
>>>> >>>>> solve
>>>> >>>>>
>>>> >>>>> I'm also attaching a zip file which contains the simple PETSc
>>>> >script
>>>> >>>>> that loads the binary matrix/vector as well as two test cases, if
>>>> >you guys
>>>> >>>>> are interested in trying it out. It only works if you have PETSc
>>>> >configured
>>>> >>>>> with complex numbers.
>>>> >>>>>
>>>> >>>>> Thanks
>>>> >>>>>
>>>> >>>>> Justin
>>>> >>>>>
>>>> >>>>> PS - A couple years ago I had asked if there was a paper/tutorial
>>>> >on
>>>> >>>>> using/tuning GAMG. Does such a thing exist today?
>>>> >>>>>
>>>> >>>>
>>>> >>>> There is a write up in the manual that is tutorial like.
>>>> >>>>
>>>> >>>>
>>>> >>>>>
>>>> >>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
>>>> ><knepley at gmail.com>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
>>>> ><jychang48 at gmail.com>
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
>>>> >to
>>>> >>>>>>> solve:
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>> >>>>>>>
>>>> >>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>>>> >inverting
>>>> >>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
>>>> >equations
>>>> >>>>>>> like those in the next section.
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>> >>>>>> M-matrix to me (if all the y's are positive). This means
>>>> >>>>>> that it should be really easy to solve, and I think GAMG should
>>>> >do it.
>>>> >>>>>> You can start out just doing relaxation, like SOR, on
>>>> >>>>>> small examples.
>>>> >>>>>>
>>>> >>>>>>   Thanks,
>>>> >>>>>>
>>>> >>>>>>     Matt
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
>>>> ><knepley at gmail.com>
>>>> >>>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>> >>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> Hi all,
>>>> >>>>>>>>>
>>>> >>>>>>>>> I'm working with some folks to extract a linear system of
>>>> >equations
>>>> >>>>>>>>> from an external software package that solves power flow
>>>> >equations in
>>>> >>>>>>>>> complex form. Since that external package uses serial direct
>>>> >solvers like
>>>> >>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the same
>>>> >matrix can
>>>> >>>>>>>>> be solved in PETSc using its parallel solvers.
>>>> >>>>>>>>>
>>>> >>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>> >>>>>>>>> processes on a single node (went from solving a 300k dog
>>>> >system in 1.8
>>>> >>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
>>>> >solvers and
>>>> >>>>>>>>> preconditioners but I have never worked with complex numbers
>>>> >so I am not
>>>> >>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>> >>>>>>>>>
>>>> >>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
>>>> >(unsurprisingly). I
>>>> >>>>>>>>> believe I also tried BICG with BJACOBI and while it did
>>>> >converge it
>>>> >>>>>>>>> converged slowly. Does anyone have recommendations on how one
>>>> >would go
>>>> >>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
>>>> >was originally
>>>> >>>>>>>>> thinking about converting it to cartesian form: Declaring all
>>>> >voltages =
>>>> >>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a
>>>> >conditional
>>>> >>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
>>>> >literature that
>>>> >>>>>>>>> claim to successfully precondition power flow equations
>>>> >operate in this
>>>> >>>>>>>>> form.
>>>> >>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> 1) We really need to see the (simplified) equations
>>>> >>>>>>>>
>>>> >>>>>>>> 2) All complex equations can be converted to a system of real
>>>> >>>>>>>> equations twice as large, but this is not necessarily the best
>>>> >way to go
>>>> >>>>>>>>
>>>> >>>>>>>>  Thanks,
>>>> >>>>>>>>
>>>> >>>>>>>>     Matt
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>> Justin
>>>> >>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> --
>>>> >>>>>>>> What most experimenters take for granted before they begin
>>>> >their
>>>> >>>>>>>> experiments is infinitely more interesting than any results to
>>>> >which their
>>>> >>>>>>>> experiments lead.
>>>> >>>>>>>> -- Norbert Wiener
>>>> >>>>>>>>
>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>> >>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>> >>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> What most experimenters take for granted before they begin their
>>>> >>>>>> experiments is infinitely more interesting than any results to
>>>> >which their
>>>> >>>>>> experiments lead.
>>>> >>>>>> -- Norbert Wiener
>>>> >>>>>>
>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/
>>>> >>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>> >>>>>>
>>>> >>>>>
>>>> >>
>>>> >> --
>>>> >> What most experimenters take for granted before they begin their
>>>> >> experiments is infinitely more interesting than any results to which
>>>> >their
>>>> >> experiments lead.
>>>> >> -- Norbert Wiener
>>>> >>
>>>> >> https://www.cse.buffalo.edu/~knepley/
>>>> >> <http://www.cse.buffalo.edu/~knepley/>
>>>> >>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190202/6a33c701/attachment-0001.html>

From mfadams at lbl.gov  Sat Feb  2 05:20:07 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 2 Feb 2019 06:20:07 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAJ98EDrTL34Q3z_+gpqz95nQ-7Yx40QeOWfS08TYJtxdEPm84Q@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<0CB4DE08-A3AB-4A19-8BB0-5543347EEC4D@cc.gatech.edu>
	<CADOhEh5wskSx5QhyTPgXpdDSW07CKmWR9m7cgm+49OcP0oBfhg@mail.gmail.com>
	<CAJ98EDqeogf3pLb00KLw5z-CL03SKiX8U-SCZZdp992NQr9LSA@mail.gmail.com>
	<CADOhEh5Ghug2YB29uxUHBAdi0+vmS5XeqW1nvk-mFA5aa+-W8w@mail.gmail.com>
	<CAJ98EDrTL34Q3z_+gpqz95nQ-7Yx40QeOWfS08TYJtxdEPm84Q@mail.gmail.com>
Message-ID: <CADOhEh5FSeUVz6SMaW0eUb0HNJxqjhy2oouWqm+9vT-eZtKthQ@mail.gmail.com>

Still a good catch.
Thanks,
ps, if you set the ksp to something other than the default of preonly then
sor should work, but still not recommended.

On Sat, Feb 2, 2019 at 5:44 AM Dave May <dave.mayhem23 at gmail.com> wrote:

>
>
> On Sat, 2 Feb 2019 at 04:05, Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>> On Fri, Feb 1, 2019 at 5:32 PM Dave May <dave.mayhem23 at gmail.com> wrote:
>>
>>> I'd suggest
>>> 1) using LU on the coarse grid (you have 1 application of SOR) and
>>>
>>
>> Ah, good catch!
>>
>> By setting every level KSP/PC explicitly you seemed to have overroad the
>> coarse grid LU solver that is set by default in GAMG.
>>
>
> Not really - there was an explicit option provided specifying the coarse
> pc to use SOR.
>
>
>
>>
>>> 2) do more Richardson iterations on each level (you have the default of
>>> 2). Try 4 or 6
>>>
>>> Thanks,
>>>   Dave
>>>
>>> On Fri, 1 Feb 2019 at 22:18, Mark Adams via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> We do need the equations and discretization, this not a Laplacian or
>>>> something is not working out-of-th-box.
>>>>
>>>> For pc_gamg_agg_nsmooths 0 you want  -pc_gamg_square_graph 10
>>>>
>>>> The info output snows the size of the systems on each level and the
>>>> number of non-zeros. with pc_gamg_agg_nsmooths 0 in 2D it courses very
>>>> slowly. You want a reduction factor between grids of say 4 - 10 between
>>>> grids.
>>>>
>>>> And here is shorthand for r/sor on all levels:
>>>>
>>>> -mg_levels_ksp_type richardson
>>>> -mg_levels_pc_type sor
>>>>
>>>> On Fri, Feb 1, 2019 at 4:07 PM Isaac, Tobin G via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> - Resuggesting a look at the max/min eigenvalues for as big a system
>>>>> as possible.
>>>>> - I like doing fgmres with a monitored krylov method to see if the
>>>>> smoother is behaving very differently on the fine vs intermediate vs coarse
>>>>> levels
>>>>>
>>>>>
>>>>> On February 1, 2019 3:53:21 PM EST, Justin Chang via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> >I tried these options:
>>>>> >
>>>>> >-ksp_view
>>>>> >-ksp_monitor_true_residual
>>>>> >-ksp_type gmres
>>>>> >-ksp_max_it 50
>>>>> >-pc_type gamg
>>>>> >-mg_coarse_pc_type sor
>>>>> >-mg_levels_1_ksp_type richardson
>>>>> >-mg_levels_2_ksp_type richardson
>>>>> >-mg_levels_3_ksp_type richardson
>>>>> >-mg_levels_4_ksp_type richardson
>>>>> >-mg_levels_1_pc_type sor
>>>>> >-mg_levels_2_pc_type sor
>>>>> >-mg_levels_3_pc_type sor
>>>>> >-mg_levels_4_pc_type sor
>>>>> >
>>>>> >And still have a non-converging solution:
>>>>> >
>>>>> >  0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>>>>> >1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>>>>> >  1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>>>>> >1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>>>>> >  2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>>>>> >1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>>>>> >  3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>>>>> >1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>>>>> >  4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>>>>> >1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>>>>> >  5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>>>>> >1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>>>>> >  6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>>>>> >1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>>>>> >  7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>>>>> >1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>>>>> >  8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>>>>> >1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>>>>> >  9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>>>>> >1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>>>>> > 10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>>>>> >1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>>>>> > 11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>>>>> >1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>>>>> > 12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>>>>> >1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>>>>> > 13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>>>>> >1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>>>>> > 14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>>>>> >1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>>>>> > 15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>>>>> >1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>>>>> > 16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>>>>> >1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>>>>> > 17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>>>>> >1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>>>>> > 18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>>>>> >1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>>>>> > 19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>>>>> >1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>>>>> > 20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>>>>> >1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>>>>> > 21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>>>>> >1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>>>>> > 22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>>>>> >1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>>>>> > 23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>>>>> >1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>>>>> > 24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>>>>> >1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>>>>> > 25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>>>>> >1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>>>>> > 26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>>>>> >1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>>>>> > 27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>>>>> >1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>>>>> > 28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>>>>> >1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>>>>> > 29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>>>>> >1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>>>>> > 30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>>>>> >1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>>>>> > 31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>>>>> >1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>>>>> > 32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>>>>> >1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>>>>> > 33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>>>>> >1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>>>>> > 34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>>>>> >1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>>>>> > 35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>>>>> >1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>>>>> > 36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>>>>> >1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>>>>> > 37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>>>>> >1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>>>>> > 38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>>>>> >1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>>>>> > 39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>>>>> >1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>>>>> > 40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>>>>> >1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>>>>> > 41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>>>>> >1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>>>>> > 42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>>>>> >1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>>>>> > 43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>>>>> >1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>>>>> > 44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>>>>> >1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>>>>> > 45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>>>>> >1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>>>>> > 46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>>>>> >1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>>>>> > 47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>>>>> >1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>>>>> > 48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>>>>> >1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>>>>> > 49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>>>>> >1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>>>>> > 50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>>>>> >1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>>>>> >KSP Object: 1 MPI processes
>>>>> >  type: gmres
>>>>> >restart=30, using Classical (unmodified) Gram-Schmidt
>>>>> Orthogonalization
>>>>> >with no iterative refinement
>>>>> >    happy breakdown tolerance 1e-30
>>>>> >  maximum iterations=50, initial guess is zero
>>>>> >  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >  left preconditioning
>>>>> >  using PRECONDITIONED norm type for convergence test
>>>>> >PC Object: 1 MPI processes
>>>>> >  type: gamg
>>>>> >    type is MULTIPLICATIVE, levels=5 cycles=v
>>>>> >      Cycles per PCApply=1
>>>>> >      Using externally compute Galerkin coarse grid matrices
>>>>> >      GAMG specific options
>>>>> >      Threshold for dropping small values in graph on each level =
>>>>>  0.
>>>>> >0.   0.
>>>>> >        Threshold scaling factor for each level not specified = 1.
>>>>> >        AGG specific options
>>>>> >          Symmetric graph false
>>>>> >          Number of levels to square graph 1
>>>>> >          Number smoothing steps 1
>>>>> >        Complexity:    grid = 1.31821
>>>>> >  Coarse grid solver -- level -------------------------------
>>>>> >    KSP Object: (mg_coarse_) 1 MPI processes
>>>>> >      type: preonly
>>>>> >      maximum iterations=10000, initial guess is zero
>>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >      left preconditioning
>>>>> >      using NONE norm type for convergence test
>>>>> >    PC Object: (mg_coarse_) 1 MPI processes
>>>>> >      type: sor
>>>>> >    type = local_symmetric, iterations = 1, local iterations = 1,
>>>>> omega
>>>>> >= 1.
>>>>> >      linear system matrix = precond matrix:
>>>>> >      Mat Object: 1 MPI processes
>>>>> >        type: seqaij
>>>>> >        rows=17, cols=17
>>>>> >        total: nonzeros=163, allocated nonzeros=163
>>>>> >        total number of mallocs used during MatSetValues calls =0
>>>>> >          using I-node routines: found 12 nodes, limit used is 5
>>>>> >  Down solver (pre-smoother) on level 1
>>>>> -------------------------------
>>>>> >    KSP Object: (mg_levels_1_) 1 MPI processes
>>>>> >      type: richardson
>>>>> >        damping factor=1.
>>>>> >      maximum iterations=2, nonzero initial guess
>>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >      left preconditioning
>>>>> >      using NONE norm type for convergence test
>>>>> >    PC Object: (mg_levels_1_) 1 MPI processes
>>>>> >      type: sor
>>>>> >    type = local_symmetric, iterations = 1, local iterations = 1,
>>>>> omega
>>>>> >= 1.
>>>>> >      linear system matrix = precond matrix:
>>>>> >      Mat Object: 1 MPI processes
>>>>> >        type: seqaij
>>>>> >        rows=100, cols=100
>>>>> >        total: nonzeros=1240, allocated nonzeros=1240
>>>>> >        total number of mallocs used during MatSetValues calls =0
>>>>> >          not using I-node routines
>>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>> >  Down solver (pre-smoother) on level 2
>>>>> -------------------------------
>>>>> >    KSP Object: (mg_levels_2_) 1 MPI processes
>>>>> >      type: richardson
>>>>> >        damping factor=1.
>>>>> >      maximum iterations=2, nonzero initial guess
>>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >      left preconditioning
>>>>> >      using NONE norm type for convergence test
>>>>> >    PC Object: (mg_levels_2_) 1 MPI processes
>>>>> >      type: sor
>>>>> >    type = local_symmetric, iterations = 1, local iterations = 1,
>>>>> omega
>>>>> >= 1.
>>>>> >      linear system matrix = precond matrix:
>>>>> >      Mat Object: 1 MPI processes
>>>>> >        type: seqaij
>>>>> >        rows=537, cols=537
>>>>> >        total: nonzeros=5291, allocated nonzeros=5291
>>>>> >        total number of mallocs used during MatSetValues calls =0
>>>>> >          not using I-node routines
>>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>> >  Down solver (pre-smoother) on level 3
>>>>> -------------------------------
>>>>> >    KSP Object: (mg_levels_3_) 1 MPI processes
>>>>> >      type: richardson
>>>>> >        damping factor=1.
>>>>> >      maximum iterations=2, nonzero initial guess
>>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >      left preconditioning
>>>>> >      using NONE norm type for convergence test
>>>>> >    PC Object: (mg_levels_3_) 1 MPI processes
>>>>> >      type: sor
>>>>> >    type = local_symmetric, iterations = 1, local iterations = 1,
>>>>> omega
>>>>> >= 1.
>>>>> >      linear system matrix = precond matrix:
>>>>> >      Mat Object: 1 MPI processes
>>>>> >        type: seqaij
>>>>> >        rows=1541, cols=1541
>>>>> >        total: nonzeros=8039, allocated nonzeros=8039
>>>>> >        total number of mallocs used during MatSetValues calls =0
>>>>> >          not using I-node routines
>>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>> >  Down solver (pre-smoother) on level 4
>>>>> -------------------------------
>>>>> >    KSP Object: (mg_levels_4_) 1 MPI processes
>>>>> >      type: richardson
>>>>> >        damping factor=1.
>>>>> >      maximum iterations=2, nonzero initial guess
>>>>> >      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>> >      left preconditioning
>>>>> >      using NONE norm type for convergence test
>>>>> >    PC Object: (mg_levels_4_) 1 MPI processes
>>>>> >      type: sor
>>>>> >    type = local_symmetric, iterations = 1, local iterations = 1,
>>>>> omega
>>>>> >= 1.
>>>>> >      linear system matrix = precond matrix:
>>>>> >      Mat Object: 1 MPI processes
>>>>> >        type: seqaij
>>>>> >        rows=8541, cols=8541
>>>>> >        total: nonzeros=46299, allocated nonzeros=46299
>>>>> >        total number of mallocs used during MatSetValues calls =0
>>>>> >          using I-node routines: found 5464 nodes, limit used is 5
>>>>> >  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>> >  linear system matrix = precond matrix:
>>>>> >  Mat Object: 1 MPI processes
>>>>> >    type: seqaij
>>>>> >    rows=8541, cols=8541
>>>>> >    total: nonzeros=46299, allocated nonzeros=46299
>>>>> >    total number of mallocs used during MatSetValues calls =0
>>>>> >      using I-node routines: found 5464 nodes, limit used is 5
>>>>> >
>>>>> >Am I doing this right? Did I miss anything?
>>>>> >
>>>>> >
>>>>> >
>>>>> >On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>>>>> >wrote:
>>>>> >
>>>>> >> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>>>>> >wrote:
>>>>> >>
>>>>> >>> Hi Mark,
>>>>> >>>
>>>>> >>> 1) So with these options:
>>>>> >>>
>>>>> >>> -ksp_type gmres
>>>>> >>> -ksp_rtol 1e-15
>>>>> >>> -ksp_monitor_true_residual
>>>>> >>> -ksp_converged_reason
>>>>> >>> -pc_type bjacobi
>>>>> >>>
>>>>> >>> This is what I get:
>>>>> >>>
>>>>> >>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid
>>>>> norm
>>>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>> >>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid
>>>>> norm
>>>>> >>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>>> >>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid
>>>>> norm
>>>>> >>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>>> >>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid
>>>>> norm
>>>>> >>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>>> >>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid
>>>>> norm
>>>>> >>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>>> >>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid
>>>>> norm
>>>>> >>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>>> >>>
>>>>> >>> 2) With richardson/sor:
>>>>> >>>
>>>>> >>
>>>>> >> Okay, its looks like Richardson/SOR solves this just fine. You can
>>>>> >use
>>>>> >> this as the smoother for GAMG instead
>>>>> >> of Cheby/Jacobi, and probably see better results on the larger
>>>>> >problems.
>>>>> >>
>>>>> >>   Matt
>>>>> >>
>>>>> >>
>>>>> >>> -ksp_type richardson
>>>>> >>> -ksp_rtol 1e-15
>>>>> >>> -ksp_monitor_true_residual
>>>>> >>> -pc_type sor
>>>>> >>>
>>>>> >>> This is what I get:
>>>>> >>>
>>>>> >>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid
>>>>> norm
>>>>> >>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>> >>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid
>>>>> norm
>>>>> >>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>>> >>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid
>>>>> norm
>>>>> >>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>>> >>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid
>>>>> norm
>>>>> >>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>>> >>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid
>>>>> norm
>>>>> >>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>>> >>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid
>>>>> norm
>>>>> >>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>>> >>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid
>>>>> norm
>>>>> >>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>>> >>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid
>>>>> norm
>>>>> >>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>>> >>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid
>>>>> norm
>>>>> >>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>>> >>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid
>>>>> norm
>>>>> >>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>>> >>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid
>>>>> norm
>>>>> >>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>>> >>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid
>>>>> norm
>>>>> >>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>>> >>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid
>>>>> norm
>>>>> >>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>>> >>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid
>>>>> norm
>>>>> >>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>>> >>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid
>>>>> norm
>>>>> >>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>>> >>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid
>>>>> norm
>>>>> >>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>>> >>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid
>>>>> norm
>>>>> >>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>>> >>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid
>>>>> norm
>>>>> >>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>>> >>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid
>>>>> norm
>>>>> >>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>>> >>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid
>>>>> norm
>>>>> >>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>>> >>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid
>>>>> norm
>>>>> >>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>>> >>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid
>>>>> norm
>>>>> >>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>>> >>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid
>>>>> norm
>>>>> >>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>>> >>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid
>>>>> norm
>>>>> >>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>>> >>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid
>>>>> norm
>>>>> >>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>>> >>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid
>>>>> norm
>>>>> >>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>>> >>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid
>>>>> norm
>>>>> >>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>>> >>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid
>>>>> norm
>>>>> >>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>>> >>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid
>>>>> norm
>>>>> >>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>>> >>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid
>>>>> norm
>>>>> >>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>>> >>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid
>>>>> norm
>>>>> >>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>>> >>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid
>>>>> norm
>>>>> >>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>>> >>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid
>>>>> norm
>>>>> >>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>>> >>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid
>>>>> norm
>>>>> >>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>>> >>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid
>>>>> norm
>>>>> >>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>>> >>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid
>>>>> norm
>>>>> >>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>>> >>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid
>>>>> norm
>>>>> >>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>>> >>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid
>>>>> norm
>>>>> >>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>>> >>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid
>>>>> norm
>>>>> >>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>>> >>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid
>>>>> norm
>>>>> >>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>>> >>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid
>>>>> norm
>>>>> >>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>>> >>>
>>>>> >>> 3) And lastly with chebyshev/jacobi:
>>>>> >>>
>>>>> >>> -ksp_type chebyshev
>>>>> >>> -ksp_rtol 1e-15
>>>>> >>> -ksp_monitor_true_residual
>>>>> >>> -pc_type jacobi
>>>>> >>>
>>>>> >>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid
>>>>> norm
>>>>> >>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>>> >>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid
>>>>> norm
>>>>> >>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>>> >>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid
>>>>> norm
>>>>> >>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>>> >>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid
>>>>> norm
>>>>> >>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>>> >>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid
>>>>> norm
>>>>> >>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>>> >>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid
>>>>> norm
>>>>> >>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>>> >>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid
>>>>> norm
>>>>> >>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>>> >>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid
>>>>> norm
>>>>> >>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>>> >>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid
>>>>> norm
>>>>> >>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>>> >>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid
>>>>> norm
>>>>> >>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>>> >>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid
>>>>> norm
>>>>> >>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>>> >>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid
>>>>> norm
>>>>> >>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>>> >>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid
>>>>> norm
>>>>> >>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>>> >>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid
>>>>> norm
>>>>> >>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>>> >>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid
>>>>> norm
>>>>> >>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>>> >>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid
>>>>> norm
>>>>> >>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>>> >>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid
>>>>> norm
>>>>> >>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>>> >>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid
>>>>> norm
>>>>> >>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>>> >>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid
>>>>> norm
>>>>> >>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>>> >>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid
>>>>> norm
>>>>> >>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>>> >>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid
>>>>> norm
>>>>> >>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>>> >>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid
>>>>> norm
>>>>> >>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>>> >>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid
>>>>> norm
>>>>> >>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>>> >>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid
>>>>> norm
>>>>> >>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>>> >>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid
>>>>> norm
>>>>> >>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>>> >>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid
>>>>> norm
>>>>> >>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>>> >>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid
>>>>> norm
>>>>> >>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>>> >>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid
>>>>> norm
>>>>> >>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>>> >>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid
>>>>> norm
>>>>> >>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>>> >>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid
>>>>> norm
>>>>> >>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>>> >>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid
>>>>> norm
>>>>> >>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>>> >>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid
>>>>> norm
>>>>> >>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>>> >>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid
>>>>> norm
>>>>> >>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>>> >>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid
>>>>> norm
>>>>> >>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>>> >>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid
>>>>> norm
>>>>> >>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>>> >>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid
>>>>> norm
>>>>> >>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>>> >>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid
>>>>> norm
>>>>> >>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>>> >>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid
>>>>> norm
>>>>> >>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>>> >>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid
>>>>> norm
>>>>> >>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>>> >>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid
>>>>> norm
>>>>> >>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>>> >>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid
>>>>> norm
>>>>> >>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>>> >>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid
>>>>> norm
>>>>> >>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>>> >>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid
>>>>> norm
>>>>> >>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>>> >>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid
>>>>> norm
>>>>> >>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>>> >>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid
>>>>> norm
>>>>> >>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>>> >>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid
>>>>> norm
>>>>> >>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>>> >>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid
>>>>> norm
>>>>> >>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>>> >>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid
>>>>> norm
>>>>> >>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>>> >>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid
>>>>> norm
>>>>> >>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>>> >>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid
>>>>> norm
>>>>> >>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>>> >>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid
>>>>> norm
>>>>> >>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>>> >>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid
>>>>> norm
>>>>> >>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>>> >>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid
>>>>> norm
>>>>> >>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>>> >>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid
>>>>> norm
>>>>> >>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>>> >>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid
>>>>> norm
>>>>> >>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>>> >>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid
>>>>> norm
>>>>> >>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>>> >>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid
>>>>> norm
>>>>> >>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>>> >>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid
>>>>> norm
>>>>> >>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>>> >>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid
>>>>> norm
>>>>> >>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>>> >>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid
>>>>> norm
>>>>> >>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>>> >>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid
>>>>> norm
>>>>> >>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>>> >>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid
>>>>> norm
>>>>> >>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>>> >>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid
>>>>> norm
>>>>> >>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>>> >>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid
>>>>> norm
>>>>> >>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>>> >>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid
>>>>> norm
>>>>> >>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>>> >>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid
>>>>> norm
>>>>> >>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>>> >>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid
>>>>> norm
>>>>> >>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>>>> >>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>>> >>>
>>>>> >>> Looks like neither of these ksp/pc combos are good? I also tried
>>>>> >>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all.
>>>>> >Here's the
>>>>> >>> GAMG info I was able to grep from the out-of-box params:
>>>>> >>>
>>>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>>> >>> nnz/row (ave)=5, np=1
>>>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>>>> >threshold
>>>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>>>> >>> min=2.394056e-02 PC=jacobi
>>>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>>>>> >active
>>>>> >>> pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 5.21674 nnz ave. (N=1541)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>>>> >>> min=6.783380e-02 PC=jacobi
>>>>> >>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 9.85289 nnz ave. (N=537)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>>>> >>> min=5.974776e-02 PC=jacobi
>>>>> >>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>>>>> >active
>>>>> >>> pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 12.4 nnz ave. (N=100)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>>>> >>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>>>> >>> min=4.842076e-01 PC=jacobi
>>>>> >>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1
>>>>> >active pes
>>>>> >>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>>> >>>
>>>>> >>> And the one with pc_gamg_agg_nsmooths 0
>>>>> >>>
>>>>> >>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>>> >>> nnz/row (ave)=5, np=1
>>>>> >>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with
>>>>> >threshold
>>>>> >>> 0., 5.42079 nnz ave. (N=8541)
>>>>> >>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>>>>> >active
>>>>> >>> pes
>>>>> >>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with
>>>>> >threshold
>>>>> >>> 0., 3.07398 nnz ave. (N=1541)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 3.02211 nnz ave. (N=814)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 3.00434 nnz ave. (N=461)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 3. nnz ave. (N=290)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 3. nnz ave. (N=197)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 2.98425 nnz ave. (N=127)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 2.97561 nnz ave. (N=82)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1
>>>>> >active pes
>>>>> >>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> >0.,
>>>>> >>> 2.9697 nnz ave. (N=66)
>>>>> >>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>>>> >>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1
>>>>> >active pes
>>>>> >>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>>> >>>
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> Justin
>>>>> >>>
>>>>> >>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>> >>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>> Both GAMG and ILU are nice and dandy for this,
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny
>>>>> system
>>>>> >and
>>>>> >>>> converge it way down, say rtol = 1.e-12. See which one is better
>>>>> in
>>>>> >the
>>>>> >>>> early iteration and pick it. It would be nice to check that it
>>>>> >solves the
>>>>> >>>> problem ...
>>>>> >>>>
>>>>> >>>> The residual drops about 5 orders in the first iteration and then
>>>>> >>>> flatlines. That is very bad. Check that the smoother can actually
>>>>> >solve the
>>>>> >>>> problem.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>> but as soon as I look at a bigger system, like a network with
>>>>> 8500
>>>>> >>>>> buses, the out-of-box gamg craps out. I am not sure where to
>>>>> start
>>>>> >when it
>>>>> >>>>> comes to tuning GAMG.
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>> First try -pc_gamg_nsmooths 0
>>>>> >>>>
>>>>> >>>> You can run with -info and grep on GAMG to see some diagnostic
>>>>> >output.
>>>>> >>>> Eigen estimates are fragile in practice and with this parameter
>>>>> and
>>>>> >SOR
>>>>> >>>> there are no eigen estimates needed. The max eigen values for all
>>>>> >levels
>>>>> >>>> should be between say 2 (or less) and say 3-4. Much higher is a
>>>>> >sign of a
>>>>> >>>> problem.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>> Attached is the ksp monitor/view output for gamg on the
>>>>> >unsuccessful
>>>>> >>>>> solve
>>>>> >>>>>
>>>>> >>>>> I'm also attaching a zip file which contains the simple PETSc
>>>>> >script
>>>>> >>>>> that loads the binary matrix/vector as well as two test cases, if
>>>>> >you guys
>>>>> >>>>> are interested in trying it out. It only works if you have PETSc
>>>>> >configured
>>>>> >>>>> with complex numbers.
>>>>> >>>>>
>>>>> >>>>> Thanks
>>>>> >>>>>
>>>>> >>>>> Justin
>>>>> >>>>>
>>>>> >>>>> PS - A couple years ago I had asked if there was a paper/tutorial
>>>>> >on
>>>>> >>>>> using/tuning GAMG. Does such a thing exist today?
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>> There is a write up in the manual that is tutorial like.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley
>>>>> ><knepley at gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>>
>>>>> >>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang
>>>>> ><jychang48 at gmail.com>
>>>>> >>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>> Here's IMHO the simplest explanation of the equations I'm
>>>>> trying
>>>>> >to
>>>>> >>>>>>> solve:
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>> >>>>>>>
>>>>> >>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>>>>> >inverting
>>>>> >>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve
>>>>> >equations
>>>>> >>>>>>> like those in the next section.
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like
>>>>> an
>>>>> >>>>>> M-matrix to me (if all the y's are positive). This means
>>>>> >>>>>> that it should be really easy to solve, and I think GAMG should
>>>>> >do it.
>>>>> >>>>>> You can start out just doing relaxation, like SOR, on
>>>>> >>>>>> small examples.
>>>>> >>>>>>
>>>>> >>>>>>   Thanks,
>>>>> >>>>>>
>>>>> >>>>>>     Matt
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley
>>>>> ><knepley at gmail.com>
>>>>> >>>>>>> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>> >>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>>> Hi all,
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I'm working with some folks to extract a linear system of
>>>>> >equations
>>>>> >>>>>>>>> from an external software package that solves power flow
>>>>> >equations in
>>>>> >>>>>>>>> complex form. Since that external package uses serial direct
>>>>> >solvers like
>>>>> >>>>>>>>> KLU from suitesparse, I want a proof-of-concept where the
>>>>> same
>>>>> >matrix can
>>>>> >>>>>>>>> be solved in PETSc using its parallel solvers.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>> >>>>>>>>> processes on a single node (went from solving a 300k dog
>>>>> >system in 1.8
>>>>> >>>>>>>>> seconds to 1.5 seconds). However I want to use iterative
>>>>> >solvers and
>>>>> >>>>>>>>> preconditioners but I have never worked with complex numbers
>>>>> >so I am not
>>>>> >>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> So far I tried GMRES/BJACOBI and it craps out
>>>>> >(unsurprisingly). I
>>>>> >>>>>>>>> believe I also tried BICG with BJACOBI and while it did
>>>>> >converge it
>>>>> >>>>>>>>> converged slowly. Does anyone have recommendations on how one
>>>>> >would go
>>>>> >>>>>>>>> about preconditioning PETSc matrices with complex numbers? I
>>>>> >was originally
>>>>> >>>>>>>>> thinking about converting it to cartesian form: Declaring all
>>>>> >voltages =
>>>>> >>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like
>>>>> a
>>>>> >conditional
>>>>> >>>>>>>>> arctan(imaginary/real) because all the papers I've seen in
>>>>> >literature that
>>>>> >>>>>>>>> claim to successfully precondition power flow equations
>>>>> >operate in this
>>>>> >>>>>>>>> form.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> 1) We really need to see the (simplified) equations
>>>>> >>>>>>>>
>>>>> >>>>>>>> 2) All complex equations can be converted to a system of real
>>>>> >>>>>>>> equations twice as large, but this is not necessarily the best
>>>>> >way to go
>>>>> >>>>>>>>
>>>>> >>>>>>>>  Thanks,
>>>>> >>>>>>>>
>>>>> >>>>>>>>     Matt
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>> Justin
>>>>> >>>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> --
>>>>> >>>>>>>> What most experimenters take for granted before they begin
>>>>> >their
>>>>> >>>>>>>> experiments is infinitely more interesting than any results to
>>>>> >which their
>>>>> >>>>>>>> experiments lead.
>>>>> >>>>>>>> -- Norbert Wiener
>>>>> >>>>>>>>
>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> >>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>> >>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> What most experimenters take for granted before they begin their
>>>>> >>>>>> experiments is infinitely more interesting than any results to
>>>>> >which their
>>>>> >>>>>> experiments lead.
>>>>> >>>>>> -- Norbert Wiener
>>>>> >>>>>>
>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> >>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>
>>>>> >> --
>>>>> >> What most experimenters take for granted before they begin their
>>>>> >> experiments is infinitely more interesting than any results to which
>>>>> >their
>>>>> >> experiments lead.
>>>>> >> -- Norbert Wiener
>>>>> >>
>>>>> >> https://www.cse.buffalo.edu/~knepley/
>>>>> >> <http://www.cse.buffalo.edu/~knepley/>
>>>>> >>
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190202/a5018bc4/attachment-0001.html>

From moritz.huck at rwth-aachen.de  Sat Feb  2 12:08:40 2019
From: moritz.huck at rwth-aachen.de (Moritz Huck)
Date: Sat, 2 Feb 2019 19:08:40 +0100
Subject: [petsc-users] [TimeStepping] Eventhandler
Message-ID: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>

Hi,
I am trying to use SetEventHandler.
I hav eencountered two "problems":

1. after an event is encountered the next time step is needlessly small 
(h=1e-6), even if a don't change anything in the function (for testing).


2. the integration does not continue at the event encounter
e.g. my eventfunction is g(t,u)=t-1,
the event is found in the step from t= 0.9 to 1.1 at tevent=1
but the next step is not 1+dt but it is 1.1 + dt
I would expect the next step to be 1.+dt

Best Regards,
Moritz
	

From knepley at gmail.com  Sat Feb  2 14:41:44 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 2 Feb 2019 15:41:44 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
Message-ID: <CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>

The coarse grid is getting set to SOR rather than LU.

   Matt

On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:

> I tried these options:
>
> -ksp_view
> -ksp_monitor_true_residual
> -ksp_type gmres
> -ksp_max_it 50
> -pc_type gamg
> -mg_coarse_pc_type sor
> -mg_levels_1_ksp_type richardson
> -mg_levels_2_ksp_type richardson
> -mg_levels_3_ksp_type richardson
> -mg_levels_4_ksp_type richardson
> -mg_levels_1_pc_type sor
> -mg_levels_2_pc_type sor
> -mg_levels_3_pc_type sor
> -mg_levels_4_pc_type sor
>
> And still have a non-converging solution:
>
>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=5 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.
>   0.   0.
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>         Complexity:    grid = 1.31821
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=17, cols=17
>         total: nonzeros=163, allocated nonzeros=163
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 12 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=100, cols=100
>         total: nonzeros=1240, allocated nonzeros=1240
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=537, cols=537
>         total: nonzeros=5291, allocated nonzeros=5291
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=1541, cols=1541
>         total: nonzeros=8039, allocated nonzeros=8039
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=8541, cols=8541
>         total: nonzeros=46299, allocated nonzeros=46299
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 5464 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=8541, cols=8541
>     total: nonzeros=46299, allocated nonzeros=46299
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 5464 nodes, limit used is 5
>
> Am I doing this right? Did I miss anything?
>
>
>
> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Hi Mark,
>>>
>>> 1) So with these options:
>>>
>>> -ksp_type gmres
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -ksp_converged_reason
>>> -pc_type bjacobi
>>>
>>> This is what I get:
>>>
>>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>
>>> 2) With richardson/sor:
>>>
>>
>> Okay, its looks like Richardson/SOR solves this just fine. You can use
>> this as the smoother for GAMG instead
>> of Cheby/Jacobi, and probably see better results on the larger problems.
>>
>>   Matt
>>
>>
>>> -ksp_type richardson
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -pc_type sor
>>>
>>> This is what I get:
>>>
>>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>
>>> 3) And lastly with chebyshev/jacobi:
>>>
>>> -ksp_type chebyshev
>>> -ksp_rtol 1e-15
>>> -ksp_monitor_true_residual
>>> -pc_type jacobi
>>>
>>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>
>>> Looks like neither of these ksp/pc combos are good? I also tried
>>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
>>> GAMG info I was able to grep from the out-of-box params:
>>>
>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> nnz/row (ave)=5, np=1
>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>> 0., 5.42079 nnz ave. (N=8541)
>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>> min=2.394056e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 5.21674 nnz ave. (N=1541)
>>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>> min=6.783380e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 9.85289 nnz ave. (N=537)
>>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>> min=5.974776e-02 PC=jacobi
>>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 12.4 nnz ave. (N=100)
>>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>> min=4.842076e-01 PC=jacobi
>>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
>>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>
>>> And the one with pc_gamg_agg_nsmooths 0
>>>
>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>> nnz/row (ave)=5, np=1
>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>> 0., 5.42079 nnz ave. (N=8541)
>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold
>>> 0., 3.07398 nnz ave. (N=1541)
>>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 3.02211 nnz ave. (N=814)
>>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 3.00434 nnz ave. (N=461)
>>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 3. nnz ave. (N=290)
>>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 3. nnz ave. (N=197)
>>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
>>> pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 2.98425 nnz ave. (N=127)
>>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 2.97561 nnz ave. (N=82)
>>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>> 2.9697 nnz ave. (N=66)
>>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
>>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>
>>>
>>> Thanks,
>>> Justin
>>>
>>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>>
>>>>
>>>>> Both GAMG and ILU are nice and dandy for this,
>>>>>
>>>>
>>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
>>>> converge it way down, say rtol = 1.e-12. See which one is better in the
>>>> early iteration and pick it. It would be nice to check that it solves the
>>>> problem ...
>>>>
>>>> The residual drops about 5 orders in the first iteration and then
>>>> flatlines. That is very bad. Check that the smoother can actually solve the
>>>> problem.
>>>>
>>>>
>>>>> but as soon as I look at a bigger system, like a network with 8500
>>>>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>>>>> comes to tuning GAMG.
>>>>>
>>>>
>>>> First try -pc_gamg_nsmooths 0
>>>>
>>>> You can run with -info and grep on GAMG to see some diagnostic output.
>>>> Eigen estimates are fragile in practice and with this parameter and SOR
>>>> there are no eigen estimates needed. The max eigen values for all levels
>>>> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
>>>> problem.
>>>>
>>>>
>>>>> Attached is the ksp monitor/view output for gamg on the unsuccessful
>>>>> solve
>>>>>
>>>>> I'm also attaching a zip file which contains the simple PETSc script
>>>>> that loads the binary matrix/vector as well as two test cases, if you guys
>>>>> are interested in trying it out. It only works if you have PETSc configured
>>>>> with complex numbers.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Justin
>>>>>
>>>>> PS - A couple years ago I had asked if there was a paper/tutorial on
>>>>> using/tuning GAMG. Does such a thing exist today?
>>>>>
>>>>
>>>> There is a write up in the manual that is tutorial like.
>>>>
>>>>
>>>>>
>>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>>>>>> solve:
>>>>>>>
>>>>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>>>
>>>>>>> Right now we're just trying to solve eq(5) (in section 1), inverting
>>>>>>> the linear Y-bus matrix. Eventually we have to be able to solve equations
>>>>>>> like those in the next section.
>>>>>>>
>>>>>>
>>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>>>> M-matrix to me (if all the y's are positive). This means
>>>>>> that it should be really easy to solve, and I think GAMG should do
>>>>>> it. You can start out just doing relaxation, like SOR, on
>>>>>> small examples.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I'm working with some folks to extract a linear system of
>>>>>>>>> equations from an external software package that solves power flow
>>>>>>>>> equations in complex form. Since that external package uses serial direct
>>>>>>>>> solvers like KLU from suitesparse, I want a proof-of-concept where the same
>>>>>>>>> matrix can be solved in PETSc using its parallel solvers.
>>>>>>>>>
>>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>>>>>> processes on a single node (went from solving a 300k dog system in 1.8
>>>>>>>>> seconds to 1.5 seconds). However I want to use iterative solvers and
>>>>>>>>> preconditioners but I have never worked with complex numbers so I am not
>>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>>>>>>
>>>>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>>>>> form.
>>>>>>>>>
>>>>>>>>
>>>>>>>> 1) We really need to see the (simplified) equations
>>>>>>>>
>>>>>>>> 2) All complex equations can be converted to a system of real
>>>>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>>>>
>>>>>>>>  Thanks,
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Justin
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190202/a37d5102/attachment-0001.html>

From hongzhang at anl.gov  Sat Feb  2 18:59:42 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Sun, 3 Feb 2019 00:59:42 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
Message-ID: <2E10F391-7326-48D8-AA99-1BAEB0250039@anl.gov>

Hi Moritz,

Are you using adaptive time stepping or fixed time stepping? When a fixed stepsize is used, the next time step after an event will be set to the remaining part of the time step automatically. In your example, presumably the stepsize is dt = 0.2, the trajectory should look like
t=0.9
t=1.0 (event time)
t=1.1
t=1.3
...

So a very small stepsize after an event is expected if the event occurs near the end of the original time step. 

Hong (Mr.)

> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi,
> I am trying to use SetEventHandler.
> I hav eencountered two "problems":
> 
> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
> 
> 
> 2. the integration does not continue at the event encounter
> e.g. my eventfunction is g(t,u)=t-1,
> the event is found in the step from t= 0.9 to 1.1 at tevent=1
> but the next step is not 1+dt but it is 1.1 + dt
> I would expect the next step to be 1.+dt
> 
> Best Regards,
> Moritz
> 	


From bsmith at mcs.anl.gov  Sat Feb  2 19:43:51 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 3 Feb 2019 01:43:51 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
Message-ID: <D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>



> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi,
> I am trying to use SetEventHandler.
> I hav eencountered two "problems":
> 
> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
> 
> 
> 2. the integration does not continue at the event encounter
> e.g. my eventfunction is g(t,u)=t-1,
> the event is found in the step from t= 0.9 to 1.1 at tevent=1
> but the next step is not 1+dt but it is 1.1 + dt
> I would expect the next step to be 1.+dt

     Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?

   Thanks

   Barry

> 
> Best Regards,
> Moritz
> 	


From moritz.huck at rwth-aachen.de  Sun Feb  3 04:45:27 2019
From: moritz.huck at rwth-aachen.de (Moritz Huck)
Date: Sun, 3 Feb 2019 11:45:27 +0100
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <2E10F391-7326-48D8-AA99-1BAEB0250039@anl.gov>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<2E10F391-7326-48D8-AA99-1BAEB0250039@anl.gov>
Message-ID: <9fe2ca28-8bad-1a16-9f77-4baff378c56e@rwth-aachen.de>

Hi Hong,

I am using adaptive timestepping (ARKIMEX, fully implicit).

Best Regards,

Moritz

Am 03.02.19 um 01:59 schrieb Zhang, Hong:
> Hi Moritz,
>
> Are you using adaptive time stepping or fixed time stepping? When a fixed stepsize is used, the next time step after an event will be set to the remaining part of the time step automatically. In your example, presumably the stepsize is dt = 0.2, the trajectory should look like
> t=0.9
> t=1.0 (event time)
> t=1.1
> t=1.3
> ...
>
> So a very small stepsize after an event is expected if the event occurs near the end of the original time step.
>
> Hong (Mr.)
>
>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi,
>> I am trying to use SetEventHandler.
>> I hav eencountered two "problems":
>>
>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>
>>
>> 2. the integration does not continue at the event encounter
>> e.g. my eventfunction is g(t,u)=t-1,
>> the event is found in the step from t= 0.9 to 1.1 at tevent=1
>> but the next step is not 1+dt but it is 1.1 + dt
>> I would expect the next step to be 1.+dt
>>
>> Best Regards,
>> Moritz
>> 	

From moritz.huck at rwth-aachen.de  Sun Feb  3 06:08:43 2019
From: moritz.huck at rwth-aachen.de (Moritz Huck)
Date: Sun, 3 Feb 2019 13:08:43 +0100
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
Message-ID: <b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>

Hi Barry,

src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) 
has the same behavior:

TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
TSEvent: iter 1 - Stepping forward as no event detected in interval 
[3.61 - 4.07796]
TSEvent: iter 2 - Stepping forward as no event detected in interval 
[4.07796 - 4.08161]
TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
Ball hit the ground at t =? 4.08 seconds
 ????? TSAdapt basic arkimex 0:3 step? 13 accepted t=4.11?????? + 
1.000e-02 dt=1.000e-01? wlte=??? 0? wltea=?? -1 wlter=?? -1

 ????? TSAdapt basic arkimex 0:3 step? 14 accepted t=4.12?????? + 
1.000e-01 dt=5.000e-01? wlte=??? 0? wltea=?? -1 wlter=?? -1

It continues at the end of the "event interval" with the initial timestep.

Best Regards,

Moritz




Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>
>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi,
>> I am trying to use SetEventHandler.
>> I hav eencountered two "problems":
>>
>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>
>>
>> 2. the integration does not continue at the event encounter
>> e.g. my eventfunction is g(t,u)=t-1,
>> the event is found in the step from t= 0.9 to 1.1 at tevent=1
>> but the next step is not 1+dt but it is 1.1 + dt
>> I would expect the next step to be 1.+dt
>       Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>
>     Thanks
>
>     Barry
>
>> Best Regards,
>> Moritz
>> 	

From dalcinl at gmail.com  Sun Feb  3 08:32:31 2019
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 3 Feb 2019 17:32:31 +0300
Subject: [petsc-users] petsc4py for numpy array <-> MATDENSE binary
In-Reply-To: <CAOGsD9gN=J6fhp=mAmAYmupRPLD1ykritEnRMyHPxk9mkCQ_CQ@mail.gmail.com>
References: <CAOGsD9gWWTCGAJVMgBs9rgF16uAUY_XGqBqf02=xzXLz+yFsiQ@mail.gmail.com>
	<AE364A90-0831-4E85-9783-6E2B6FF95648@anl.gov>
	<CAOGsD9gN=J6fhp=mAmAYmupRPLD1ykritEnRMyHPxk9mkCQ_CQ@mail.gmail.com>
Message-ID: <CAEcYPwDV-4StTBDNENOrcoiH9YvfbnKvz-V4M1Q44Q+rre69hQ@mail.gmail.com>

On Sat, 2 Feb 2019 at 02:12, Sajid Ali <sajidsyed2021 at u.northwestern.edu>
wrote:

> The vector is essentially snapshots in time of a data array. I should
> probably store this as a 2D dense matrix of dimensions (dim_x*dim_y) *
> dim_z. Now I can pick one column at a time and use it for my TS Jacobian.
> Apologies for being a little unclear.
>
>
m, n = dim_x*dim_y, dim_z
A = PETSc.Mat.createDense([m,n], array=your_array)

-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190203/9b7ff595/attachment.html>

From hongzhang at anl.gov  Sun Feb  3 10:04:20 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Sun, 3 Feb 2019 16:04:20 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
Message-ID: <A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>

ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :

8 TS dt 0.5 time 3.11
9 TS dt 0.5 time 3.61
TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
9 TS dt 0.467955 time 3.61 (r)
TSEvent: iter 1 - Stepping forward as no event detected in interval [3.61 - 4.07796]
10 TS dt 0.00365179 time 4.07796
TSEvent: iter 2 - Stepping forward as no event detected in interval [4.07796 - 4.08161]
11 TS dt 2.5403e-05 time 4.08161
TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
Ball hit the ground at t =  4.08 seconds
12 TS dt 0.0283673 time 4.08163
13 TS dt 0.01 time 4.11
14 TS dt 0.1 time 4.12

The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.

Hong (Mr.)

> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi Barry,
> 
> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
> 
> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
> TSEvent: iter 1 - Stepping forward as no event detected in interval [3.61 - 4.07796]
> TSEvent: iter 2 - Stepping forward as no event detected in interval [4.07796 - 4.08161]
> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
> Ball hit the ground at t =  4.08 seconds
>       TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
> 
>       TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
> 
> It continues at the end of the "event interval" with the initial timestep.
> 
> Best Regards,
> 
> Moritz
> 
> 
> 
> 
> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>> 
>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi,
>>> I am trying to use SetEventHandler.
>>> I hav eencountered two "problems":
>>> 
>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>> 
>>> 
>>> 2. the integration does not continue at the event encounter
>>> e.g. my eventfunction is g(t,u)=t-1,
>>> the event is found in the step from t= 0.9 to 1.1 at tevent=1
>>> but the next step is not 1+dt but it is 1.1 + dt
>>> I would expect the next step to be 1.+dt
>>      Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>> 
>>    Thanks
>> 
>>    Barry
>> 
>>> Best Regards,
>>> Moritz
>>> 	


From moritz.huck at rwth-aachen.de  Sun Feb  3 10:45:29 2019
From: moritz.huck at rwth-aachen.de (Moritz Huck)
Date: Sun, 3 Feb 2019 17:45:29 +0100
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
Message-ID: <2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>

Hi,

I see I am sorry, I misinterpreted the output -ts_adapt_monitor.

You are right point 2 is not an issue.

I there a way to let the time stepping continue with the last time step 
instead of the initial time step?

Best Regards,

Moritz

Am 03.02.19 um 17:04 schrieb Zhang, Hong:
> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>
> 8 TS dt 0.5 time 3.11
> 9 TS dt 0.5 time 3.61
> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
> 9 TS dt 0.467955 time 3.61 (r)
> TSEvent: iter 1 - Stepping forward as no event detected in interval [3.61 - 4.07796]
> 10 TS dt 0.00365179 time 4.07796
> TSEvent: iter 2 - Stepping forward as no event detected in interval [4.07796 - 4.08161]
> 11 TS dt 2.5403e-05 time 4.08161
> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
> Ball hit the ground at t =  4.08 seconds
> 12 TS dt 0.0283673 time 4.08163
> 13 TS dt 0.01 time 4.11
> 14 TS dt 0.1 time 4.12
>
> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>
> Hong (Mr.)
>
>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi Barry,
>>
>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>
>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>> TSEvent: iter 1 - Stepping forward as no event detected in interval [3.61 - 4.07796]
>> TSEvent: iter 2 - Stepping forward as no event detected in interval [4.07796 - 4.08161]
>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
>> Ball hit the ground at t =  4.08 seconds
>>        TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>>        TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>> It continues at the end of the "event interval" with the initial timestep.
>>
>> Best Regards,
>>
>> Moritz
>>
>>
>>
>>
>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>
>>>> Hi,
>>>> I am trying to use SetEventHandler.
>>>> I hav eencountered two "problems":
>>>>
>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>
>>>>
>>>> 2. the integration does not continue at the event encounter
>>>> e.g. my eventfunction is g(t,u)=t-1,
>>>> the event is found in the step from t= 0.9 to 1.1 at tevent=1
>>>> but the next step is not 1+dt but it is 1.1 + dt
>>>> I would expect the next step to be 1.+dt
>>>       Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>
>>>     Thanks
>>>
>>>     Barry
>>>
>>>> Best Regards,
>>>> Moritz
>>>> 	

From edoardo.alinovi at gmail.com  Sun Feb  3 08:27:57 2019
From: edoardo.alinovi at gmail.com (Edoardo alinovi)
Date: Sun, 3 Feb 2019 15:27:57 +0100
Subject: [petsc-users] Ksp Initial residual norm
Message-ID: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>

Dear all,

I have a very simple question for you. How can I get the initial residual
from a ksp solver? I can easily extract the final (preconditioned) residual
norm  but  when I try to use to do the same with the initial residual using:

kspInitialResidual()

PETSc tells me that this feature is not supported. I am actually doing it
applying the definition of residual "by hand", but I am wondering if
somenthing already built-in is available or not.

Thank you very much!
------

Edoardo Alinovi, Ph.D.

DICCA, Scuola Politecnica,
Universita' degli Studi di Genova,
1, via Montallegro,
16145 Genova, Italy

Email: edoardo.alinovi at dicca.unige.it
Tel: +39 010 353 2540
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190203/1da84e8b/attachment.html>

From bsmith at mcs.anl.gov  Sun Feb  3 13:03:24 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 3 Feb 2019 19:03:24 +0000
Subject: [petsc-users] Ksp Initial residual norm
In-Reply-To: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
References: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
Message-ID: <9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>


   If you use the default KSP convergence tests (which almost everyone does) the initial residual norm is stored in ksp->rnorm0. Thus to access it you need to include the private header file that defines the KSP object kspimpl.h so something like 

#include <petsc/private/kspimpl.h> 

   PetscErrorCode KSPGetInitialResidualNorm(KSP ksp,PetscReal *norm0)
  {
   *norm0 = ksp->rnorm0;
   return 0;
  }

should work.

   Barry




   

> On Feb 3, 2019, at 8:27 AM, Edoardo alinovi via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear all,
> 
> I have a very simple question for you. How can I get the initial residual from a ksp solver? I can easily extract the final (preconditioned) residual norm  but  when I try to use to do the same with the initial residual using:
> 
> kspInitialResidual()
> 
> PETSc tells me that this feature is not supported. I am actually doing it applying the definition of residual "by hand", but I am wondering if somenthing already built-in is available or not. 
> 
> Thank you very much! 
> ------
> 
> Edoardo Alinovi, Ph.D.
> 
> DICCA, Scuola Politecnica,
> Universita' degli Studi di Genova,
> 1, via Montallegro,
> 16145 Genova, Italy
> 
> Email: edoardo.alinovi at dicca.unige.it
> Tel: +39 010 353 2540
> 
> 
> 


From edoardo.alinovi at gmail.com  Sun Feb  3 13:16:49 2019
From: edoardo.alinovi at gmail.com (Edoardo alinovi)
Date: Sun, 3 Feb 2019 20:16:49 +0100
Subject: [petsc-users] Ksp Initial residual norm
In-Reply-To: <9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>
References: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
	<9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>
Message-ID: <CADmAu6JgPcMf+K0_MDZeMVaRYv8FkXvv-QYkMOHaLit588EfVA@mail.gmail.com>

Thank you very much Barry for the suggestion.

Unfortunately, I am using Fortran and not C++ . Do you have an equivalent
trick in this case? :)
------

Edoardo Alinovi, Ph.D.

DICCA, Scuola Politecnica,
Universita' degli Studi di Genova,
1, via Montallegro,
16145 Genova, Italy

Email: edoardo.alinovi at dicca.unige.it
Tel: +39 010 353 2540





Il giorno dom 3 feb 2019 alle ore 20:03 Smith, Barry F. <bsmith at mcs.anl.gov>
ha scritto:

>
>    If you use the default KSP convergence tests (which almost everyone
> does) the initial residual norm is stored in ksp->rnorm0. Thus to access it
> you need to include the private header file that defines the KSP object
> kspimpl.h so something like
>
> #include <petsc/private/kspimpl.h>
>
>    PetscErrorCode KSPGetInitialResidualNorm(KSP ksp,PetscReal *norm0)
>   {
>    *norm0 = ksp->rnorm0;
>    return 0;
>   }
>
> should work.
>
>    Barry
>
>
>
>
>
>
> > On Feb 3, 2019, at 8:27 AM, Edoardo alinovi via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Dear all,
> >
> > I have a very simple question for you. How can I get the initial
> residual from a ksp solver? I can easily extract the final (preconditioned)
> residual norm  but  when I try to use to do the same with the initial
> residual using:
> >
> > kspInitialResidual()
> >
> > PETSc tells me that this feature is not supported. I am actually doing
> it applying the definition of residual "by hand", but I am wondering if
> somenthing already built-in is available or not.
> >
> > Thank you very much!
> > ------
> >
> > Edoardo Alinovi, Ph.D.
> >
> > DICCA, Scuola Politecnica,
> > Universita' degli Studi di Genova,
> > 1, via Montallegro,
> > 16145 Genova, Italy
> >
> > Email: edoardo.alinovi at dicca.unige.it
> > Tel: +39 010 353 2540
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190203/32b3f73e/attachment.html>

From bsmith at mcs.anl.gov  Sun Feb  3 13:24:35 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 3 Feb 2019 19:24:35 +0000
Subject: [petsc-users] Ksp Initial residual norm
In-Reply-To: <CADmAu6JgPcMf+K0_MDZeMVaRYv8FkXvv-QYkMOHaLit588EfVA@mail.gmail.com>
References: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
	<9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>
	<CADmAu6JgPcMf+K0_MDZeMVaRYv8FkXvv-QYkMOHaLit588EfVA@mail.gmail.com>
Message-ID: <E77AA6DD-C2FA-4CCC-B1D8-4D23397413C6@mcs.anl.gov>



> On Feb 3, 2019, at 1:16 PM, Edoardo alinovi <edoardo.alinovi at gmail.com> wrote:
> 
> Thank you very much Barry for the suggestion. 
> 
> Unfortunately, I am using Fortran and not C++ . Do you have an equivalent trick in this case? :)

   You would need to provide this function as a C function (in its own little .c file) and then call it from your Fortran code. 

   Barry

   There is no way to access the data structure directly from Fortran.

> ------
> 
> Edoardo Alinovi, Ph.D.
> 
> DICCA, Scuola Politecnica,
> Universita' degli Studi di Genova,
> 1, via Montallegro,
> 16145 Genova, Italy
> 
> Email: edoardo.alinovi at dicca.unige.it
> Tel: +39 010 353 2540
> 
> 
> 
> 
> 
> Il giorno dom 3 feb 2019 alle ore 20:03 Smith, Barry F. <bsmith at mcs.anl.gov> ha scritto:
> 
>    If you use the default KSP convergence tests (which almost everyone does) the initial residual norm is stored in ksp->rnorm0. Thus to access it you need to include the private header file that defines the KSP object kspimpl.h so something like 
> 
> #include <petsc/private/kspimpl.h> 
> 
>    PetscErrorCode KSPGetInitialResidualNorm(KSP ksp,PetscReal *norm0)
>   {
>    *norm0 = ksp->rnorm0;
>    return 0;
>   }
> 
> should work.
> 
>    Barry
> 
> 
> 
> 
> 
> 
> > On Feb 3, 2019, at 8:27 AM, Edoardo alinovi via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Dear all,
> > 
> > I have a very simple question for you. How can I get the initial residual from a ksp solver? I can easily extract the final (preconditioned) residual norm  but  when I try to use to do the same with the initial residual using:
> > 
> > kspInitialResidual()
> > 
> > PETSc tells me that this feature is not supported. I am actually doing it applying the definition of residual "by hand", but I am wondering if somenthing already built-in is available or not. 
> > 
> > Thank you very much! 
> > ------
> > 
> > Edoardo Alinovi, Ph.D.
> > 
> > DICCA, Scuola Politecnica,
> > Universita' degli Studi di Genova,
> > 1, via Montallegro,
> > 16145 Genova, Italy
> > 
> > Email: edoardo.alinovi at dicca.unige.it
> > Tel: +39 010 353 2540
> > 
> > 
> > 
> 


From knepley at gmail.com  Sun Feb  3 15:40:46 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 3 Feb 2019 16:40:46 -0500
Subject: [petsc-users] Ksp Initial residual norm
In-Reply-To: <E77AA6DD-C2FA-4CCC-B1D8-4D23397413C6@mcs.anl.gov>
References: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
	<9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>
	<CADmAu6JgPcMf+K0_MDZeMVaRYv8FkXvv-QYkMOHaLit588EfVA@mail.gmail.com>
	<E77AA6DD-C2FA-4CCC-B1D8-4D23397413C6@mcs.anl.gov>
Message-ID: <CAMYG4GnofYcVM+emC9sgNmwykyXY++ytjdPCEAbZjsH9dJgQdg@mail.gmail.com>

On Sun, Feb 3, 2019 at 2:24 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> > On Feb 3, 2019, at 1:16 PM, Edoardo alinovi <edoardo.alinovi at gmail.com>
> wrote:
> >
> > Thank you very much Barry for the suggestion.
> >
> > Unfortunately, I am using Fortran and not C++ . Do you have an
> equivalent trick in this case? :)
>
>    You would need to provide this function as a C function (in its own
> little .c file) and then call it from your Fortran code.
>

I think you can also use:


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPGetResidualHistory.html

    Matt


>    Barry
>
>    There is no way to access the data structure directly from Fortran.
>
> > ------
> >
> > Edoardo Alinovi, Ph.D.
> >
> > DICCA, Scuola Politecnica,
> > Universita' degli Studi di Genova,
> > 1, via Montallegro,
> > 16145 Genova, Italy
> >
> > Email: edoardo.alinovi at dicca.unige.it
> > Tel: +39 010 353 2540
> >
> >
> >
> >
> >
> > Il giorno dom 3 feb 2019 alle ore 20:03 Smith, Barry F. <
> bsmith at mcs.anl.gov> ha scritto:
> >
> >    If you use the default KSP convergence tests (which almost everyone
> does) the initial residual norm is stored in ksp->rnorm0. Thus to access it
> you need to include the private header file that defines the KSP object
> kspimpl.h so something like
> >
> > #include <petsc/private/kspimpl.h>
> >
> >    PetscErrorCode KSPGetInitialResidualNorm(KSP ksp,PetscReal *norm0)
> >   {
> >    *norm0 = ksp->rnorm0;
> >    return 0;
> >   }
> >
> > should work.
> >
> >    Barry
> >
> >
> >
> >
> >
> >
> > > On Feb 3, 2019, at 8:27 AM, Edoardo alinovi via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > >
> > > Dear all,
> > >
> > > I have a very simple question for you. How can I get the initial
> residual from a ksp solver? I can easily extract the final (preconditioned)
> residual norm  but  when I try to use to do the same with the initial
> residual using:
> > >
> > > kspInitialResidual()
> > >
> > > PETSc tells me that this feature is not supported. I am actually doing
> it applying the definition of residual "by hand", but I am wondering if
> somenthing already built-in is available or not.
> > >
> > > Thank you very much!
> > > ------
> > >
> > > Edoardo Alinovi, Ph.D.
> > >
> > > DICCA, Scuola Politecnica,
> > > Universita' degli Studi di Genova,
> > > 1, via Montallegro,
> > > 16145 Genova, Italy
> > >
> > > Email: edoardo.alinovi at dicca.unige.it
> > > Tel: +39 010 353 2540
> > >
> > >
> > >
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190203/0a68fca2/attachment.html>

From bsmith at mcs.anl.gov  Sun Feb  3 16:06:37 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 3 Feb 2019 22:06:37 +0000
Subject: [petsc-users] Ksp Initial residual norm
In-Reply-To: <CAMYG4GnofYcVM+emC9sgNmwykyXY++ytjdPCEAbZjsH9dJgQdg@mail.gmail.com>
References: <CADmAu6JVUX5a-CXJ3TNYm69NgdOj3N+wHBJ_R4NLL61CaZENCw@mail.gmail.com>
	<9F36BA6C-B6EB-466E-B16B-D75AEC2D3FBD@anl.gov>
	<CADmAu6JgPcMf+K0_MDZeMVaRYv8FkXvv-QYkMOHaLit588EfVA@mail.gmail.com>
	<E77AA6DD-C2FA-4CCC-B1D8-4D23397413C6@mcs.anl.gov>
	<CAMYG4GnofYcVM+emC9sgNmwykyXY++ytjdPCEAbZjsH9dJgQdg@mail.gmail.com>
Message-ID: <354C2DC9-0A00-4D2F-B167-63641A16717D@mcs.anl.gov>


  Oh yes. Better than my suggestion since it requires no coding.\

  Barry


> On Feb 3, 2019, at 3:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sun, Feb 3, 2019 at 2:24 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> > On Feb 3, 2019, at 1:16 PM, Edoardo alinovi <edoardo.alinovi at gmail.com> wrote:
> > 
> > Thank you very much Barry for the suggestion. 
> > 
> > Unfortunately, I am using Fortran and not C++ . Do you have an equivalent trick in this case? :)
> 
>    You would need to provide this function as a C function (in its own little .c file) and then call it from your Fortran code. 
> 
> I think you can also use:
> 
>   https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPGetResidualHistory.html
> 
>     Matt
>  
>    Barry
> 
>    There is no way to access the data structure directly from Fortran.
> 
> > ------
> > 
> > Edoardo Alinovi, Ph.D.
> > 
> > DICCA, Scuola Politecnica,
> > Universita' degli Studi di Genova,
> > 1, via Montallegro,
> > 16145 Genova, Italy
> > 
> > Email: edoardo.alinovi at dicca.unige.it
> > Tel: +39 010 353 2540
> > 
> > 
> > 
> > 
> > 
> > Il giorno dom 3 feb 2019 alle ore 20:03 Smith, Barry F. <bsmith at mcs.anl.gov> ha scritto:
> > 
> >    If you use the default KSP convergence tests (which almost everyone does) the initial residual norm is stored in ksp->rnorm0. Thus to access it you need to include the private header file that defines the KSP object kspimpl.h so something like 
> > 
> > #include <petsc/private/kspimpl.h> 
> > 
> >    PetscErrorCode KSPGetInitialResidualNorm(KSP ksp,PetscReal *norm0)
> >   {
> >    *norm0 = ksp->rnorm0;
> >    return 0;
> >   }
> > 
> > should work.
> > 
> >    Barry
> > 
> > 
> > 
> > 
> > 
> > 
> > > On Feb 3, 2019, at 8:27 AM, Edoardo alinovi via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > Dear all,
> > > 
> > > I have a very simple question for you. How can I get the initial residual from a ksp solver? I can easily extract the final (preconditioned) residual norm  but  when I try to use to do the same with the initial residual using:
> > > 
> > > kspInitialResidual()
> > > 
> > > PETSc tells me that this feature is not supported. I am actually doing it applying the definition of residual "by hand", but I am wondering if somenthing already built-in is available or not. 
> > > 
> > > Thank you very much! 
> > > ------
> > > 
> > > Edoardo Alinovi, Ph.D.
> > > 
> > > DICCA, Scuola Politecnica,
> > > Universita' degli Studi di Genova,
> > > 1, via Montallegro,
> > > 16145 Genova, Italy
> > > 
> > > Email: edoardo.alinovi at dicca.unige.it
> > > Tel: +39 010 353 2540
> > > 
> > > 
> > > 
> > 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From Moritz.Huck at rwth-aachen.de  Mon Feb  4 01:56:21 2019
From: Moritz.Huck at rwth-aachen.de (Huck, Moritz)
Date: Mon, 4 Feb 2019 07:56:21 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>,
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
Message-ID: <660c205547424a4db43951a5bc1fce38@rwth-aachen.de>

Hi,
yes there is one.
My system has a set of very different time contants.
At some events the fast one may be fully developped and only the slower ones are of concern after the event (depending on the postevent).
If the last step is not an option, setting it manually might be helpfull since I generall know which timeconstants will be relevant (TSSetTime step seems not to have an effect during the postevent call).

Moritz
________________________________________
Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
Gesendet: Montag, 4. Februar 2019 05:09:24
An: Huck, Moritz; Zhang, Hong
Betreff: RE: [petsc-users] [TimeStepping] Eventhandler

Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.

Shri

-----Original Message-----
From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Moritz Huck via petsc-users
Sent: Sunday, February 3, 2019 10:45 AM
To: Zhang, Hong <hongzhang at anl.gov>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] [TimeStepping] Eventhandler

Hi,

I see I am sorry, I misinterpreted the output -ts_adapt_monitor.

You are right point 2 is not an issue.

I there a way to let the time stepping continue with the last time step instead of the initial time step?

Best Regards,

Moritz

Am 03.02.19 um 17:04 schrieb Zhang, Hong:
> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>
> 8 TS dt 0.5 time 3.11
> 9 TS dt 0.5 time 3.61
> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
> 9 TS dt 0.467955 time 3.61 (r)
> TSEvent: iter 1 - Stepping forward as no event detected in interval
> [3.61 - 4.07796]
> 10 TS dt 0.00365179 time 4.07796
> TSEvent: iter 2 - Stepping forward as no event detected in interval
> [4.07796 - 4.08161]
> 11 TS dt 2.5403e-05 time 4.08161
> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
> Ball hit the ground at t =  4.08 seconds
> 12 TS dt 0.0283673 time 4.08163
> 13 TS dt 0.01 time 4.11
> 14 TS dt 0.1 time 4.12
>
> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>
> Hong (Mr.)
>
>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi Barry,
>>
>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>
>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>> [3.61 - 4.07796]
>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>> [4.07796 - 4.08161]
>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>> iterations Ball hit the ground at t =  4.08 seconds
>>        TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>>        TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>> It continues at the end of the "event interval" with the initial timestep.
>>
>> Best Regards,
>>
>> Moritz
>>
>>
>>
>>
>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>
>>>> Hi,
>>>> I am trying to use SetEventHandler.
>>>> I hav eencountered two "problems":
>>>>
>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>
>>>>
>>>> 2. the integration does not continue at the event encounter e.g. my
>>>> eventfunction is g(t,u)=t-1, the event is found in the step from t=
>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is 1.1
>>>> + dt I would expect the next step to be 1.+dt
>>>       Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>
>>>     Thanks
>>>
>>>     Barry
>>>
>>>> Best Regards,
>>>> Moritz
>>>>

From Moritz.Huck at rwth-aachen.de  Mon Feb  4 02:23:39 2019
From: Moritz.Huck at rwth-aachen.de (Huck, Moritz)
Date: Mon, 4 Feb 2019 08:23:39 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>,
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>,
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
Message-ID: <696e9a067c1e44f4bef4416df568efff@rwth-aachen.de>

as an added issue.
The steps to complete the time interval after event detection, seem to be done with the intial step, this leads to man steps with a very small step for me.
(e.g. my initial step is 1e-6 but the event at 1000s is detected in [980..1070] is nedd to to take 70e6 steps with dt=1e-6)

________________________________________
Von: Huck, Moritz
Gesendet: Montag, 4. Februar 2019 08:56:21
An: Abhyankar, Shrirang G; Zhang, Hong; petsc-users at mcs.anl.gov
Betreff: AW: [petsc-users] [TimeStepping] Eventhandler

Hi,
yes there is one.
My system has a set of very different time contants.
At some events the fast one may be fully developped and only the slower ones are of concern after the event (depending on the postevent).
If the last step is not an option, setting it manually might be helpfull since I generall know which timeconstants will be relevant (TSSetTime step seems not to have an effect during the postevent call).

Moritz
________________________________________
Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
Gesendet: Montag, 4. Februar 2019 05:09:24
An: Huck, Moritz; Zhang, Hong
Betreff: RE: [petsc-users] [TimeStepping] Eventhandler

Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.

Shri

-----Original Message-----
From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Moritz Huck via petsc-users
Sent: Sunday, February 3, 2019 10:45 AM
To: Zhang, Hong <hongzhang at anl.gov>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] [TimeStepping] Eventhandler

Hi,

I see I am sorry, I misinterpreted the output -ts_adapt_monitor.

You are right point 2 is not an issue.

I there a way to let the time stepping continue with the last time step instead of the initial time step?

Best Regards,

Moritz

Am 03.02.19 um 17:04 schrieb Zhang, Hong:
> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>
> 8 TS dt 0.5 time 3.11
> 9 TS dt 0.5 time 3.61
> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
> 9 TS dt 0.467955 time 3.61 (r)
> TSEvent: iter 1 - Stepping forward as no event detected in interval
> [3.61 - 4.07796]
> 10 TS dt 0.00365179 time 4.07796
> TSEvent: iter 2 - Stepping forward as no event detected in interval
> [4.07796 - 4.08161]
> 11 TS dt 2.5403e-05 time 4.08161
> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
> Ball hit the ground at t =  4.08 seconds
> 12 TS dt 0.0283673 time 4.08163
> 13 TS dt 0.01 time 4.11
> 14 TS dt 0.1 time 4.12
>
> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>
> Hong (Mr.)
>
>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi Barry,
>>
>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>
>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>> [3.61 - 4.07796]
>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>> [4.07796 - 4.08161]
>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>> iterations Ball hit the ground at t =  4.08 seconds
>>        TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>>        TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>
>> It continues at the end of the "event interval" with the initial timestep.
>>
>> Best Regards,
>>
>> Moritz
>>
>>
>>
>>
>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>
>>>> Hi,
>>>> I am trying to use SetEventHandler.
>>>> I hav eencountered two "problems":
>>>>
>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>
>>>>
>>>> 2. the integration does not continue at the event encounter e.g. my
>>>> eventfunction is g(t,u)=t-1, the event is found in the step from t=
>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is 1.1
>>>> + dt I would expect the next step to be 1.+dt
>>>       Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>
>>>     Thanks
>>>
>>>     Barry
>>>
>>>> Best Regards,
>>>> Moritz
>>>>

From amansaxena.iitg at gmail.com  Mon Feb  4 05:03:54 2019
From: amansaxena.iitg at gmail.com (Aman Saxena)
Date: Mon, 4 Feb 2019 16:33:54 +0530
Subject: [petsc-users] Queries related to snes options
Message-ID: <CAHpK7ExLAv0x-syY1p2N7LiG-Ln5JLbZm5+_dBYXzNdd9_L4Jg@mail.gmail.com>

I am trying to use PETSc's TS to implement implicit time-stepping, for
1d-Euler Equation discretized using 5th order WENO schemes. I have
following queries related to using some of the snes options::

-- What does -ts_max_snes_failure means and what does the non-linear solver
do once it diverges to ensure convergence?

-- If I do not specify ?-snes_fd? or ?-snes_mf?, does by default it uses
the jacobian I have coded instead of building it using Finite-differences?

-- As ?-snes_fd ? replaces my routine to calculate the jacobian with
petsc?s fd routine, does it stores the jacobian in the same Matrix that I
pass to my hand-coded routine. It might lead to some inconsistency as my
actual scheme involves stencils width equals 3 while I am approximating the
Jacobian using first order scheme which requires stencils width equals 1.
So I have used another ?dmda? with sw = 1, to initialize the matrix using
?DMCreateMatrix()? while fd needs sw = 3.

-- I am not able to use ?-snes_type test? it gives me the following error
?[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
http://www.mcs.anl.gov/petsc/documentation/installation.html#external
 [0]PETSC ERROR: Unable to find requested SNES type test?

-- What exactly "PCMAT" do? Do I have to supply matrix other than in
RHSJacobian for PCMAT?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/634bd2b5/attachment.html>

From mfadams at lbl.gov  Mon Feb  4 09:37:39 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 4 Feb 2019 10:37:39 -0500
Subject: [petsc-users] Queries related to snes options
In-Reply-To: <CAHpK7ExLAv0x-syY1p2N7LiG-Ln5JLbZm5+_dBYXzNdd9_L4Jg@mail.gmail.com>
References: <CAHpK7ExLAv0x-syY1p2N7LiG-Ln5JLbZm5+_dBYXzNdd9_L4Jg@mail.gmail.com>
Message-ID: <CADOhEh5gYUbcJb9szj5=Hmc-7VPXdUKNuLQgObVOn=+U1RpMMA@mail.gmail.com>

On Mon, Feb 4, 2019 at 6:05 AM Aman Saxena via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I am trying to use PETSc's TS to implement implicit time-stepping, for
> 1d-Euler Equation discretized using 5th order WENO schemes. I have
> following queries related to using some of the snes options::
>
> -- What does -ts_max_snes_failure means and what does the non-linear
> solver do once it diverges to ensure convergence?
>

You want to look at the manual but I think this is the number of nonlinear
solver failures that it will try to recover from before quitting.


>
> -- If I do not specify ?-snes_fd? or ?-snes_mf?, does by default it uses
> the jacobian I have coded instead of building it using Finite-differences?
>

test it but if you give it a Jacobian I would hope that it uses it!

>
> -- As ?-snes_fd ? replaces my routine to calculate the jacobian with
> petsc?s fd routine, does it stores the jacobian in the same Matrix that I
> pass to my hand-coded routine. It might lead to some inconsistency as my
> actual scheme involves stencils width equals 3 while I am approximating the
> Jacobian using first order scheme which requires stencils width equals 1.
> So I have used another ?dmda? with sw = 1, to initialize the matrix using
> ?DMCreateMatrix()? while fd needs sw = 3.
>

This should be OK. Try it.


>
> -- I am not able to use ?-snes_type test? it gives me the following error
> ?[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
> http://www.mcs.anl.gov/petsc/documentation/installation.html#external
>  [0]PETSC ERROR: Unable to find requested SNES type test?
>
>
This has changed. Use -snes_test_jacobian


> -- What exactly "PCMAT" do? Do I have to supply matrix other than in
> RHSJacobian for PCMAT?
>

I'm not sure where you picked this up you don't need to worry about it:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMAT.html


>
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/42d2e37e/attachment.html>

From M.Tiberga at tudelft.nl  Mon Feb  4 10:11:32 2019
From: M.Tiberga at tudelft.nl (Marco Tiberga)
Date: Mon, 4 Feb 2019 16:11:32 +0000
Subject: [petsc-users] Doubt on how to copy a Mat into another (Fortran)
Message-ID: <63b35018005c4a88824e903f6b3f47a9@tudelft.nl>

Dear PETSc developers,

Since I am learning how to use MatCreateNest, I was looking at example ex73f90t.F90<https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex73f90t.F90.html> (I am using Fortran).

At line 297, the array of matrices to be passed to MatCreateNest is initialized with code lines as "matArray(1) = Amat".

I was surprised to find such a simple command, I thought a matrix copy should be done by a sequence like
" call MatDuplicate(Amat, MAT_SHARE_NONZERO_PATTERN,matArray(1),ierr); CHKERRA(ierr)
  call MatCopy(Amat,matArray(1), SAME_NONZERO_PATTERN,ierr); CHKERRA(ierr) "

So, I was wondering: are the two ways completely equivalent, and therefore the second unnecessarily more complex? or is the latter more robust and therefore preferable?
Is there any difference in terms of performance?

Thanks a lot for the clarification.

Best regards,
Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/6a23132a/attachment-0001.html>

From knepley at gmail.com  Mon Feb  4 10:46:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 Feb 2019 11:46:16 -0500
Subject: [petsc-users] Doubt on how to copy a Mat into another (Fortran)
In-Reply-To: <63b35018005c4a88824e903f6b3f47a9@tudelft.nl>
References: <63b35018005c4a88824e903f6b3f47a9@tudelft.nl>
Message-ID: <CAMYG4GkvETKZV5siTM=uoUh=O4PxwoSMfFOhLORR1teDbDSbeA@mail.gmail.com>

On Mon, Feb 4, 2019 at 11:16 AM Marco Tiberga via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear PETSc developers,
>
>
>
> Since I am learning how to use MatCreateNest, I was looking at example
> ex73f90t.F90
> <https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex73f90t.F90.html>
> (I am using Fortran).
>
>
>
> At line 297, the array of matrices to be passed to MatCreateNest is
> initialized with code lines as ?matArray(1) = Amat?.
>

That is a pointer. It does not create a replica in memory, as the calls
below do. Fortran is not explicit about this distinction
which can lead to confusion.

  Thanks,

    Matt


>
>
> I was surprised to find such a simple command, I thought a matrix copy
> should be done by a sequence like
>
> ? call MatDuplicate(Amat, MAT_SHARE_NONZERO_PATTERN,matArray(1),ierr);
> CHKERRA(ierr)
>
>   call MatCopy(Amat,matArray(1), SAME_NONZERO_PATTERN,ierr); CHKERRA(ierr)
> ?
>
>
>
> So, I was wondering: are the two ways completely equivalent, and therefore
> the second unnecessarily more complex? or is the latter more robust and
> therefore preferable?
>
> Is there any difference in terms of performance?
>
>
>
> Thanks a lot for the clarification.
>
>
>
> Best regards,
>
> Marco Tiberga
>
> PhD candidate
>
> Delft University of Technology
>
> Faculty of Applied Sciences
>
> Radiation Science & Technology Department
>
> Mekelweg 15, 2629 JB Delft, The Netherlands
>
> E-Mail: *m.tiberga at tudelft.nl <m.tiberga at tudelft.nl>*
>
> Website: http://www.nera.rst.tudelft.nl/
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/c763e39d/attachment.html>

From jychang48 at gmail.com  Mon Feb  4 13:56:45 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 4 Feb 2019 12:56:45 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
Message-ID: <CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>

Thanks everyone for your suggestions/feedback. So a few things:

1) When I examined larger distribution networks (~150k buses) my eigenvalue
estimates from the chebyshev method get enormous indeed. See below:

[0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
nnz/row (ave)=6, np=1
[0] PCGAMGFilterGraph():   98.5638% nnz after filtering, with threshold 0.,
6.01293 nnz ave. (N=320745)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 44797 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
min=4.639523e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
[0] PCGAMGFilterGraph():   99.9753% nnz after filtering, with threshold 0.,
7.32435 nnz ave. (N=44797)
[0] PCGAMGProlongator_AGG(): New grid 13043 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
min=9.687506e-01 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active
pes
[0] PCGAMGFilterGraph():   99.684% nnz after filtering, with threshold 0.,
22.5607 nnz ave. (N=13043)
[0] PCGAMGProlongator_AGG(): New grid 2256 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
min=6.150856e-01 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
[0] PCGAMGFilterGraph():   93.859% nnz after filtering, with threshold 0.,
79.5142 nnz ave. (N=2256)
[0] PCGAMGProlongator_AGG(): New grid 232 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
min=6.780909e-01 PC=jacobi
[0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
[0] PCGAMGFilterGraph():   99.1729% nnz after filtering, with threshold 0.,
206.379 nnz ave. (N=232)
[0] PCGAMGProlongator_AGG(): New grid 9 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
min=2.153627e-01 PC=jacobi
[0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
[0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058

2) I tried all the suggestions mentioned before: setting
-pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
iterations of richardson/sor.

3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just
solving LU on the whole system.

3b)The problem is that I can't get it to even speedup across 2 MPI
processes for my 150k bus case (~300k dofs) - and I already checked that
this problem could in theory be parallelized by setting the -ksp_max_it to
something low and observing that the KSPSolve time decreases as MPI
concurrency increases. The potential speedup is countered by the fact that
the algorithmic convergence rate blows up when either more MPI processes
are added or when I tune the block_size/overlap parameters.

Which leads me to one of two conclusions/questions:

A) Is my 300 DOF problem still too small? Do I need to look at a problem
with, say, 10 Million DOF or more to see that increasing the asm_block_size
will have better performance (despite having more KSP iterations) than a
single giant asm_block?

B) Or is there something even more sinister about my system of equations in
complex form?

Thanks,
Justin

On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com> wrote:

> The coarse grid is getting set to SOR rather than LU.
>
>    Matt
>
> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
>
>> I tried these options:
>>
>> -ksp_view
>> -ksp_monitor_true_residual
>> -ksp_type gmres
>> -ksp_max_it 50
>> -pc_type gamg
>> -mg_coarse_pc_type sor
>> -mg_levels_1_ksp_type richardson
>> -mg_levels_2_ksp_type richardson
>> -mg_levels_3_ksp_type richardson
>> -mg_levels_4_ksp_type richardson
>> -mg_levels_1_pc_type sor
>> -mg_levels_2_pc_type sor
>> -mg_levels_3_pc_type sor
>> -mg_levels_4_pc_type sor
>>
>> And still have a non-converging solution:
>>
>>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>> KSP Object: 1 MPI processes
>>   type: gmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=50, initial guess is zero
>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>   left preconditioning
>>   using PRECONDITIONED norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: gamg
>>     type is MULTIPLICATIVE, levels=5 cycles=v
>>       Cycles per PCApply=1
>>       Using externally compute Galerkin coarse grid matrices
>>       GAMG specific options
>>         Threshold for dropping small values in graph on each level =   0.
>>   0.   0.
>>         Threshold scaling factor for each level not specified = 1.
>>         AGG specific options
>>           Symmetric graph false
>>           Number of levels to square graph 1
>>           Number smoothing steps 1
>>         Complexity:    grid = 1.31821
>>   Coarse grid solver -- level -------------------------------
>>     KSP Object: (mg_coarse_) 1 MPI processes
>>       type: preonly
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>       left preconditioning
>>       using NONE norm type for convergence test
>>     PC Object: (mg_coarse_) 1 MPI processes
>>       type: sor
>>         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>>       linear system matrix = precond matrix:
>>       Mat Object: 1 MPI processes
>>         type: seqaij
>>         rows=17, cols=17
>>         total: nonzeros=163, allocated nonzeros=163
>>         total number of mallocs used during MatSetValues calls =0
>>           using I-node routines: found 12 nodes, limit used is 5
>>   Down solver (pre-smoother) on level 1 -------------------------------
>>     KSP Object: (mg_levels_1_) 1 MPI processes
>>       type: richardson
>>         damping factor=1.
>>       maximum iterations=2, nonzero initial guess
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>       left preconditioning
>>       using NONE norm type for convergence test
>>     PC Object: (mg_levels_1_) 1 MPI processes
>>       type: sor
>>         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>>       linear system matrix = precond matrix:
>>       Mat Object: 1 MPI processes
>>         type: seqaij
>>         rows=100, cols=100
>>         total: nonzeros=1240, allocated nonzeros=1240
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>   Down solver (pre-smoother) on level 2 -------------------------------
>>     KSP Object: (mg_levels_2_) 1 MPI processes
>>       type: richardson
>>         damping factor=1.
>>       maximum iterations=2, nonzero initial guess
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>       left preconditioning
>>       using NONE norm type for convergence test
>>     PC Object: (mg_levels_2_) 1 MPI processes
>>       type: sor
>>         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>>       linear system matrix = precond matrix:
>>       Mat Object: 1 MPI processes
>>         type: seqaij
>>         rows=537, cols=537
>>         total: nonzeros=5291, allocated nonzeros=5291
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>   Down solver (pre-smoother) on level 3 -------------------------------
>>     KSP Object: (mg_levels_3_) 1 MPI processes
>>       type: richardson
>>         damping factor=1.
>>       maximum iterations=2, nonzero initial guess
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>       left preconditioning
>>       using NONE norm type for convergence test
>>     PC Object: (mg_levels_3_) 1 MPI processes
>>       type: sor
>>         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>>       linear system matrix = precond matrix:
>>       Mat Object: 1 MPI processes
>>         type: seqaij
>>         rows=1541, cols=1541
>>         total: nonzeros=8039, allocated nonzeros=8039
>>         total number of mallocs used during MatSetValues calls =0
>>           not using I-node routines
>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>   Down solver (pre-smoother) on level 4 -------------------------------
>>     KSP Object: (mg_levels_4_) 1 MPI processes
>>       type: richardson
>>         damping factor=1.
>>       maximum iterations=2, nonzero initial guess
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>       left preconditioning
>>       using NONE norm type for convergence test
>>     PC Object: (mg_levels_4_) 1 MPI processes
>>       type: sor
>>         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>>       linear system matrix = precond matrix:
>>       Mat Object: 1 MPI processes
>>         type: seqaij
>>         rows=8541, cols=8541
>>         total: nonzeros=46299, allocated nonzeros=46299
>>         total number of mallocs used during MatSetValues calls =0
>>           using I-node routines: found 5464 nodes, limit used is 5
>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>   linear system matrix = precond matrix:
>>   Mat Object: 1 MPI processes
>>     type: seqaij
>>     rows=8541, cols=8541
>>     total: nonzeros=46299, allocated nonzeros=46299
>>     total number of mallocs used during MatSetValues calls =0
>>       using I-node routines: found 5464 nodes, limit used is 5
>>
>> Am I doing this right? Did I miss anything?
>>
>>
>>
>> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
>>>
>>>> Hi Mark,
>>>>
>>>> 1) So with these options:
>>>>
>>>> -ksp_type gmres
>>>> -ksp_rtol 1e-15
>>>> -ksp_monitor_true_residual
>>>> -ksp_converged_reason
>>>> -pc_type bjacobi
>>>>
>>>> This is what I get:
>>>>
>>>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>>
>>>> 2) With richardson/sor:
>>>>
>>>
>>> Okay, its looks like Richardson/SOR solves this just fine. You can use
>>> this as the smoother for GAMG instead
>>> of Cheby/Jacobi, and probably see better results on the larger problems.
>>>
>>>   Matt
>>>
>>>
>>>> -ksp_type richardson
>>>> -ksp_rtol 1e-15
>>>> -ksp_monitor_true_residual
>>>> -pc_type sor
>>>>
>>>> This is what I get:
>>>>
>>>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>>
>>>> 3) And lastly with chebyshev/jacobi:
>>>>
>>>> -ksp_type chebyshev
>>>> -ksp_rtol 1e-15
>>>> -ksp_monitor_true_residual
>>>> -pc_type jacobi
>>>>
>>>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>>
>>>> Looks like neither of these ksp/pc combos are good? I also tried
>>>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
>>>> GAMG info I was able to grep from the out-of-box params:
>>>>
>>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=5, np=1
>>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>>> 0., 5.42079 nnz ave. (N=8541)
>>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>>> min=2.394056e-02 PC=jacobi
>>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 5.21674 nnz ave. (N=1541)
>>>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>>> min=6.783380e-02 PC=jacobi
>>>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 9.85289 nnz ave. (N=537)
>>>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>>> min=5.974776e-02 PC=jacobi
>>>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 12.4 nnz ave. (N=100)
>>>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>>> min=4.842076e-01 PC=jacobi
>>>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active
>>>> pes
>>>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>>
>>>> And the one with pc_gamg_agg_nsmooths 0
>>>>
>>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>> nnz/row (ave)=5, np=1
>>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>>> 0., 5.42079 nnz ave. (N=8541)
>>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold
>>>> 0., 3.07398 nnz ave. (N=1541)
>>>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 3.02211 nnz ave. (N=814)
>>>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 3.00434 nnz ave. (N=461)
>>>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 3. nnz ave. (N=290)
>>>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 3. nnz ave. (N=197)
>>>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 2.98425 nnz ave. (N=127)
>>>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 2.97561 nnz ave. (N=82)
>>>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active
>>>> pes
>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold 0.,
>>>> 2.9697 nnz ave. (N=66)
>>>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active
>>>> pes
>>>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>>
>>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>>
>>>>>
>>>>>> Both GAMG and ILU are nice and dandy for this,
>>>>>>
>>>>>
>>>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>>>>> and converge it way down, say rtol = 1.e-12. See which one is better in the
>>>>> early iteration and pick it. It would be nice to check that it solves the
>>>>> problem ...
>>>>>
>>>>> The residual drops about 5 orders in the first iteration and then
>>>>> flatlines. That is very bad. Check that the smoother can actually solve the
>>>>> problem.
>>>>>
>>>>>
>>>>>> but as soon as I look at a bigger system, like a network with 8500
>>>>>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>>>>>> comes to tuning GAMG.
>>>>>>
>>>>>
>>>>> First try -pc_gamg_nsmooths 0
>>>>>
>>>>> You can run with -info and grep on GAMG to see some diagnostic output.
>>>>> Eigen estimates are fragile in practice and with this parameter and SOR
>>>>> there are no eigen estimates needed. The max eigen values for all levels
>>>>> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
>>>>> problem.
>>>>>
>>>>>
>>>>>> Attached is the ksp monitor/view output for gamg on the unsuccessful
>>>>>> solve
>>>>>>
>>>>>> I'm also attaching a zip file which contains the simple PETSc script
>>>>>> that loads the binary matrix/vector as well as two test cases, if you guys
>>>>>> are interested in trying it out. It only works if you have PETSc configured
>>>>>> with complex numbers.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Justin
>>>>>>
>>>>>> PS - A couple years ago I had asked if there was a paper/tutorial on
>>>>>> using/tuning GAMG. Does such a thing exist today?
>>>>>>
>>>>>
>>>>> There is a write up in the manual that is tutorial like.
>>>>>
>>>>>
>>>>>>
>>>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here's IMHO the simplest explanation of the equations I'm trying to
>>>>>>>> solve:
>>>>>>>>
>>>>>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>>>>
>>>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>>>>>>>> inverting the linear Y-bus matrix. Eventually we have to be able to solve
>>>>>>>> equations like those in the next section.
>>>>>>>>
>>>>>>>
>>>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>>>>> M-matrix to me (if all the y's are positive). This means
>>>>>>> that it should be really easy to solve, and I think GAMG should do
>>>>>>> it. You can start out just doing relaxation, like SOR, on
>>>>>>> small examples.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>
>>>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I'm working with some folks to extract a linear system of
>>>>>>>>>> equations from an external software package that solves power flow
>>>>>>>>>> equations in complex form. Since that external package uses serial direct
>>>>>>>>>> solvers like KLU from suitesparse, I want a proof-of-concept where the same
>>>>>>>>>> matrix can be solved in PETSc using its parallel solvers.
>>>>>>>>>>
>>>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>>>>>>> processes on a single node (went from solving a 300k dog system in 1.8
>>>>>>>>>> seconds to 1.5 seconds). However I want to use iterative solvers and
>>>>>>>>>> preconditioners but I have never worked with complex numbers so I am not
>>>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>>>>>>>
>>>>>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>>>>>>>>>> believe I also tried BICG with BJACOBI and while it did converge it
>>>>>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>>>>>> form.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1) We really need to see the (simplified) equations
>>>>>>>>>
>>>>>>>>> 2) All complex equations can be converted to a system of real
>>>>>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>>>>>
>>>>>>>>>  Thanks,
>>>>>>>>>
>>>>>>>>>     Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Justin
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/5331e7cc/attachment-0001.html>

From knepley at gmail.com  Mon Feb  4 14:06:49 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 Feb 2019 15:06:49 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
Message-ID: <CAMYG4G=tTTWJwRsGK4zNL2QHZ5+JJ8Jd-4yCLFkwxM4gLKNziw@mail.gmail.com>

On Mon, Feb 4, 2019 at 2:57 PM Justin Chang <jychang48 at gmail.com> wrote:

> Thanks everyone for your suggestions/feedback. So a few things:
>
> 1) When I examined larger distribution networks (~150k buses) my
> eigenvalue estimates from the chebyshev method get enormous indeed. See
> below:
>
> [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
> nnz/row (ave)=6, np=1
> [0] PCGAMGFilterGraph():   98.5638% nnz after filtering, with threshold
> 0., 6.01293 nnz ave. (N=320745)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
> min=4.639523e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active
> pes
> [0] PCGAMGFilterGraph():   99.9753% nnz after filtering, with threshold
> 0., 7.32435 nnz ave. (N=44797)
> [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
> min=9.687506e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active
> pes
> [0] PCGAMGFilterGraph():   99.684% nnz after filtering, with threshold
> 0., 22.5607 nnz ave. (N=13043)
> [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
> min=6.150856e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active
> pes
> [0] PCGAMGFilterGraph():   93.859% nnz after filtering, with threshold
> 0., 79.5142 nnz ave. (N=2256)
> [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
> min=6.780909e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active
> pes
> [0] PCGAMGFilterGraph():   99.1729% nnz after filtering, with threshold
> 0., 206.379 nnz ave. (N=232)
> [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
> min=2.153627e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
>
> 2) I tried all the suggestions mentioned before: setting
> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
> iterations of richardson/sor.
>

1) Can we see some convergence for these? LU HAS to be better than the
alternative, but you are saying that convergence still
stalls above that I think. Okay, but I do not understand how SOR was
working on the smaller problem, and cannot solve the coarse
problem. However, it could be if the interpolation is crap, which it sounds
like is true here.

Jed, does that makes sense? If so, I would think about doing some bootstrap
stuff.

   Matt


> 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just
> solving LU on the whole system.
>
> 3b)The problem is that I can't get it to even speedup across 2 MPI
> processes for my 150k bus case (~300k dofs) - and I already checked that
> this problem could in theory be parallelized by setting the -ksp_max_it to
> something low and observing that the KSPSolve time decreases as MPI
> concurrency increases. The potential speedup is countered by the fact that
> the algorithmic convergence rate blows up when either more MPI processes
> are added or when I tune the block_size/overlap parameters.
>
> Which leads me to one of two conclusions/questions:
>
> A) Is my 300 DOF problem still too small? Do I need to look at a problem
> with, say, 10 Million DOF or more to see that increasing the asm_block_size
> will have better performance (despite having more KSP iterations) than a
> single giant asm_block?
>
> B) Or is there something even more sinister about my system of equations
> in complex form?
>
> Thanks,
> Justin
>
> On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> The coarse grid is getting set to SOR rather than LU.
>>
>>    Matt
>>
>> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> I tried these options:
>>>
>>> -ksp_view
>>> -ksp_monitor_true_residual
>>> -ksp_type gmres
>>> -ksp_max_it 50
>>> -pc_type gamg
>>> -mg_coarse_pc_type sor
>>> -mg_levels_1_ksp_type richardson
>>> -mg_levels_2_ksp_type richardson
>>> -mg_levels_3_ksp_type richardson
>>> -mg_levels_4_ksp_type richardson
>>> -mg_levels_1_pc_type sor
>>> -mg_levels_2_pc_type sor
>>> -mg_levels_3_pc_type sor
>>> -mg_levels_4_pc_type sor
>>>
>>> And still have a non-converging solution:
>>>
>>>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>>> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>>> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>>>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>>> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>>>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>>> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>>>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>>> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>>>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>>> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>>>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>>> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>>>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>>> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>>>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>>> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>>>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>>> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>>>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>>> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>>>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>>> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>>>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>>> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>>>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>>> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>>>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>>> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>>>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>>> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>>>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>>> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>>>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>>> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>>>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>>> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>>>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>>> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>>>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>>> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>>>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>>> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>>>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>>> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>>>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>>> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>>>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>>> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>>>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>>> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>>>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>>> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>>>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>>> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>>>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>>> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>>>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>>> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>>>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>>> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>>>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>>> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>>>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>>> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>>>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>>> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>>>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>>> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>>>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>>> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>>>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>>> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>>>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>>> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>>>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>>> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>>>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>>> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>>>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>>> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>>>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>>> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>>>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>>> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>>>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>>> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>>>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>>> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>>>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>>> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>>>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>>> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>>>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>>> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>>>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>>> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>>>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>>> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>>>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>>> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>>> KSP Object: 1 MPI processes
>>>   type: gmres
>>>     restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>     happy breakdown tolerance 1e-30
>>>   maximum iterations=50, initial guess is zero
>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>   left preconditioning
>>>   using PRECONDITIONED norm type for convergence test
>>> PC Object: 1 MPI processes
>>>   type: gamg
>>>     type is MULTIPLICATIVE, levels=5 cycles=v
>>>       Cycles per PCApply=1
>>>       Using externally compute Galerkin coarse grid matrices
>>>       GAMG specific options
>>>         Threshold for dropping small values in graph on each level =
>>> 0.   0.   0.
>>>         Threshold scaling factor for each level not specified = 1.
>>>         AGG specific options
>>>           Symmetric graph false
>>>           Number of levels to square graph 1
>>>           Number smoothing steps 1
>>>         Complexity:    grid = 1.31821
>>>   Coarse grid solver -- level -------------------------------
>>>     KSP Object: (mg_coarse_) 1 MPI processes
>>>       type: preonly
>>>       maximum iterations=10000, initial guess is zero
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object: (mg_coarse_) 1 MPI processes
>>>       type: sor
>>>         type = local_symmetric, iterations = 1, local iterations = 1,
>>> omega = 1.
>>>       linear system matrix = precond matrix:
>>>       Mat Object: 1 MPI processes
>>>         type: seqaij
>>>         rows=17, cols=17
>>>         total: nonzeros=163, allocated nonzeros=163
>>>         total number of mallocs used during MatSetValues calls =0
>>>           using I-node routines: found 12 nodes, limit used is 5
>>>   Down solver (pre-smoother) on level 1 -------------------------------
>>>     KSP Object: (mg_levels_1_) 1 MPI processes
>>>       type: richardson
>>>         damping factor=1.
>>>       maximum iterations=2, nonzero initial guess
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object: (mg_levels_1_) 1 MPI processes
>>>       type: sor
>>>         type = local_symmetric, iterations = 1, local iterations = 1,
>>> omega = 1.
>>>       linear system matrix = precond matrix:
>>>       Mat Object: 1 MPI processes
>>>         type: seqaij
>>>         rows=100, cols=100
>>>         total: nonzeros=1240, allocated nonzeros=1240
>>>         total number of mallocs used during MatSetValues calls =0
>>>           not using I-node routines
>>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>>   Down solver (pre-smoother) on level 2 -------------------------------
>>>     KSP Object: (mg_levels_2_) 1 MPI processes
>>>       type: richardson
>>>         damping factor=1.
>>>       maximum iterations=2, nonzero initial guess
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object: (mg_levels_2_) 1 MPI processes
>>>       type: sor
>>>         type = local_symmetric, iterations = 1, local iterations = 1,
>>> omega = 1.
>>>       linear system matrix = precond matrix:
>>>       Mat Object: 1 MPI processes
>>>         type: seqaij
>>>         rows=537, cols=537
>>>         total: nonzeros=5291, allocated nonzeros=5291
>>>         total number of mallocs used during MatSetValues calls =0
>>>           not using I-node routines
>>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>>   Down solver (pre-smoother) on level 3 -------------------------------
>>>     KSP Object: (mg_levels_3_) 1 MPI processes
>>>       type: richardson
>>>         damping factor=1.
>>>       maximum iterations=2, nonzero initial guess
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object: (mg_levels_3_) 1 MPI processes
>>>       type: sor
>>>         type = local_symmetric, iterations = 1, local iterations = 1,
>>> omega = 1.
>>>       linear system matrix = precond matrix:
>>>       Mat Object: 1 MPI processes
>>>         type: seqaij
>>>         rows=1541, cols=1541
>>>         total: nonzeros=8039, allocated nonzeros=8039
>>>         total number of mallocs used during MatSetValues calls =0
>>>           not using I-node routines
>>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>>   Down solver (pre-smoother) on level 4 -------------------------------
>>>     KSP Object: (mg_levels_4_) 1 MPI processes
>>>       type: richardson
>>>         damping factor=1.
>>>       maximum iterations=2, nonzero initial guess
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object: (mg_levels_4_) 1 MPI processes
>>>       type: sor
>>>         type = local_symmetric, iterations = 1, local iterations = 1,
>>> omega = 1.
>>>       linear system matrix = precond matrix:
>>>       Mat Object: 1 MPI processes
>>>         type: seqaij
>>>         rows=8541, cols=8541
>>>         total: nonzeros=46299, allocated nonzeros=46299
>>>         total number of mallocs used during MatSetValues calls =0
>>>           using I-node routines: found 5464 nodes, limit used is 5
>>>   Up solver (post-smoother) same as down solver (pre-smoother)
>>>   linear system matrix = precond matrix:
>>>   Mat Object: 1 MPI processes
>>>     type: seqaij
>>>     rows=8541, cols=8541
>>>     total: nonzeros=46299, allocated nonzeros=46299
>>>     total number of mallocs used during MatSetValues calls =0
>>>       using I-node routines: found 5464 nodes, limit used is 5
>>>
>>> Am I doing this right? Did I miss anything?
>>>
>>>
>>>
>>> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Mark,
>>>>>
>>>>> 1) So with these options:
>>>>>
>>>>> -ksp_type gmres
>>>>> -ksp_rtol 1e-15
>>>>> -ksp_monitor_true_residual
>>>>> -ksp_converged_reason
>>>>> -pc_type bjacobi
>>>>>
>>>>> This is what I get:
>>>>>
>>>>>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>>>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>>>>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>>>>>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>>>>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>>>>>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>>>>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>>>>>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>>>>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>>>>>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>>>>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>>>>> Linear solve converged due to CONVERGED_RTOL iterations 5
>>>>>
>>>>> 2) With richardson/sor:
>>>>>
>>>>
>>>> Okay, its looks like Richardson/SOR solves this just fine. You can use
>>>> this as the smoother for GAMG instead
>>>> of Cheby/Jacobi, and probably see better results on the larger problems.
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> -ksp_type richardson
>>>>> -ksp_rtol 1e-15
>>>>> -ksp_monitor_true_residual
>>>>> -pc_type sor
>>>>>
>>>>> This is what I get:
>>>>>
>>>>>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>>>>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>>>>>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>>>>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>>>>>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>>>>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>>>>>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>>>>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>>>>>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>>>>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>>>>>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>>>>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>>>>>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>>>>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>>>>>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>>>>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>>>>>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>>>>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>>>>>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>>>>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>>>>>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>>>>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>>>>>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>>>>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>>>>>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>>>>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>>>>>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>>>>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>>>>>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>>>>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>>>>>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>>>>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>>>>>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>>>>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>>>>>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>>>>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>>>>>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>>>>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>>>>>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>>>>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>>>>>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>>>>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>>>>>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>>>>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>>>>>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>>>>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>>>>>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>>>>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>>>>>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>>>>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>>>>>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>>>>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>>>>>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>>>>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>>>>>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>>>>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>>>>>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>>>>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>>>>>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>>>>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>>>>>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>>>>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>>>>>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>>>>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>>>>>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>>>>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>>>>>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>>>>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>>>>>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>>>>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>>>>>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>>>>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>>>>>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>>>>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>>>>>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>>>>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>>>>>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>>>>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>>>>>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>>>>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>>>>>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>>>>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>>>>> Linear solve converged due to CONVERGED_RTOL iterations 40
>>>>>
>>>>> 3) And lastly with chebyshev/jacobi:
>>>>>
>>>>> -ksp_type chebyshev
>>>>> -ksp_rtol 1e-15
>>>>> -ksp_monitor_true_residual
>>>>> -pc_type jacobi
>>>>>
>>>>>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>>>>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>>>>>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>>>>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>>>>>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>>>>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>>>>>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>>>>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>>>>>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>>>>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>>>>>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>>>>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>>>>>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>>>>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>>>>>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>>>>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>>>>>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>>>>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>>>>>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>>>>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>>>>>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>>>>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>>>>>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>>>>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>>>>>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>>>>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>>>>>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>>>>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>>>>>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>>>>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>>>>>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>>>>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>>>>>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>>>>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>>>>>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>>>>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>>>>>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>>>>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>>>>>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>>>>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>>>>>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>>>>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>>>>>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>>>>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>>>>>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>>>>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>>>>>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>>>>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>>>>>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>>>>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>>>>>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>>>>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>>>>>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>>>>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>>>>>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>>>>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>>>>>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>>>>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>>>>>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>>>>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>>>>>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>>>>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>>>>>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>>>>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>>>>>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>>>>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>>>>>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>>>>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>>>>>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>>>>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>>>>>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>>>>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>>>>>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>>>>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>>>>>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>>>>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>>>>>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>>>>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>>>>>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>>>>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>>>>>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>>>>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>>>>>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>>>>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>>>>>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>>>>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>>>>>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>>>>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>>>>>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>>>>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>>>>>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>>>>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>>>>>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>>>>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>>>>>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>>>>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>>>>>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>>>>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>>>>>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>>>>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>>>>>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>>>>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>>>>>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>>>>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>>>>>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>>>>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>>>>>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>>>>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>>>>>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>>>>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>>>>>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>>>>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>>>>>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>>>>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>>>>>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>>>>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>>>>>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>>>>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>>>>>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>>>>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>>>>>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>>>>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>>>>>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>>>>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>>>>>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>>>>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>>>>>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>>>>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>>>>>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>>>>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>>>>>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>>>>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>>>>>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>>>>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>>>>> Linear solve converged due to CONVERGED_RTOL iterations 67
>>>>>
>>>>> Looks like neither of these ksp/pc combos are good? I also tried
>>>>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
>>>>> GAMG info I was able to grep from the out-of-box params:
>>>>>
>>>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>>> nnz/row (ave)=5, np=1
>>>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>>>> 0., 5.42079 nnz ave. (N=8541)
>>>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>>>>> min=2.394056e-02 PC=jacobi
>>>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1
>>>>> active pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 5.21674 nnz ave. (N=1541)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>>>>> min=6.783380e-02 PC=jacobi
>>>>> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 9.85289 nnz ave. (N=537)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>>>>> min=5.974776e-02 PC=jacobi
>>>>> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1
>>>>> active pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 12.4 nnz ave. (N=100)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>>>>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>>>>> min=4.842076e-01 PC=jacobi
>>>>> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active
>>>>> pes
>>>>> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>>>>>
>>>>> And the one with pc_gamg_agg_nsmooths 0
>>>>>
>>>>> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>>>>> nnz/row (ave)=5, np=1
>>>>> [0] PCGAMGFilterGraph(): 99.9114% nnz after filtering, with threshold
>>>>> 0., 5.42079 nnz ave. (N=8541)
>>>>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>>>>> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>>>>> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1
>>>>> active pes
>>>>> [0] PCGAMGFilterGraph(): 99.7467% nnz after filtering, with threshold
>>>>> 0., 3.07398 nnz ave. (N=1541)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>>>>> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 3.02211 nnz ave. (N=814)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>>>>> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 3.00434 nnz ave. (N=461)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>>>>> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 3. nnz ave. (N=290)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>>>>> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 3. nnz ave. (N=197)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>>>>> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 2.98425 nnz ave. (N=127)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>>>>> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 2.97561 nnz ave. (N=82)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>>>>> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active
>>>>> pes
>>>>> [0] PCGAMGFilterGraph(): 100.% nnz after filtering, with threshold
>>>>> 0., 2.9697 nnz ave. (N=66)
>>>>> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>>>>> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active
>>>>> pes
>>>>> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Justin
>>>>>
>>>>> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>> Both GAMG and ILU are nice and dandy for this,
>>>>>>>
>>>>>>
>>>>>> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system
>>>>>> and converge it way down, say rtol = 1.e-12. See which one is better in the
>>>>>> early iteration and pick it. It would be nice to check that it solves the
>>>>>> problem ...
>>>>>>
>>>>>> The residual drops about 5 orders in the first iteration and then
>>>>>> flatlines. That is very bad. Check that the smoother can actually solve the
>>>>>> problem.
>>>>>>
>>>>>>
>>>>>>> but as soon as I look at a bigger system, like a network with 8500
>>>>>>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>>>>>>> comes to tuning GAMG.
>>>>>>>
>>>>>>
>>>>>> First try -pc_gamg_nsmooths 0
>>>>>>
>>>>>> You can run with -info and grep on GAMG to see some diagnostic
>>>>>> output. Eigen estimates are fragile in practice and with this parameter and
>>>>>> SOR there are no eigen estimates needed. The max eigen values for all
>>>>>> levels should be between say 2 (or less) and say 3-4. Much higher is a sign
>>>>>> of a problem.
>>>>>>
>>>>>>
>>>>>>> Attached is the ksp monitor/view output for gamg on the unsuccessful
>>>>>>> solve
>>>>>>>
>>>>>>> I'm also attaching a zip file which contains the simple PETSc script
>>>>>>> that loads the binary matrix/vector as well as two test cases, if you guys
>>>>>>> are interested in trying it out. It only works if you have PETSc configured
>>>>>>> with complex numbers.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Justin
>>>>>>>
>>>>>>> PS - A couple years ago I had asked if there was a paper/tutorial on
>>>>>>> using/tuning GAMG. Does such a thing exist today?
>>>>>>>
>>>>>>
>>>>>> There is a write up in the manual that is tutorial like.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here's IMHO the simplest explanation of the equations I'm trying
>>>>>>>>> to solve:
>>>>>>>>>
>>>>>>>>> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>>>>>>>>>
>>>>>>>>> Right now we're just trying to solve eq(5) (in section 1),
>>>>>>>>> inverting the linear Y-bus matrix. Eventually we have to be able to solve
>>>>>>>>> equations like those in the next section.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Maybe I am reading this wrong, but the Y-bus matrix looks like an
>>>>>>>> M-matrix to me (if all the y's are positive). This means
>>>>>>>> that it should be really easy to solve, and I think GAMG should do
>>>>>>>> it. You can start out just doing relaxation, like SOR, on
>>>>>>>> small examples.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I'm working with some folks to extract a linear system of
>>>>>>>>>>> equations from an external software package that solves power flow
>>>>>>>>>>> equations in complex form. Since that external package uses serial direct
>>>>>>>>>>> solvers like KLU from suitesparse, I want a proof-of-concept where the same
>>>>>>>>>>> matrix can be solved in PETSc using its parallel solvers.
>>>>>>>>>>>
>>>>>>>>>>> I got mumps to achieve a very minor speedup across two MPI
>>>>>>>>>>> processes on a single node (went from solving a 300k dog system in 1.8
>>>>>>>>>>> seconds to 1.5 seconds). However I want to use iterative solvers and
>>>>>>>>>>> preconditioners but I have never worked with complex numbers so I am not
>>>>>>>>>>> sure what the "best" options are given PETSc's capabilities.
>>>>>>>>>>>
>>>>>>>>>>> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly).
>>>>>>>>>>> I believe I also tried BICG with BJACOBI and while it did converge it
>>>>>>>>>>> converged slowly. Does anyone have recommendations on how one would go
>>>>>>>>>>> about preconditioning PETSc matrices with complex numbers? I was originally
>>>>>>>>>>> thinking about converting it to cartesian form: Declaring all voltages =
>>>>>>>>>>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>>>>>>>>>>> arctan(imaginary/real) because all the papers I've seen in literature that
>>>>>>>>>>> claim to successfully precondition power flow equations operate in this
>>>>>>>>>>> form.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1) We really need to see the (simplified) equations
>>>>>>>>>>
>>>>>>>>>> 2) All complex equations can be converted to a system of real
>>>>>>>>>> equations twice as large, but this is not necessarily the best way to go
>>>>>>>>>>
>>>>>>>>>>  Thanks,
>>>>>>>>>>
>>>>>>>>>>     Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Justin
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/e17c08a4/attachment-0001.html>

From jed at jedbrown.org  Mon Feb  4 14:46:41 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 04 Feb 2019 13:46:41 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAMYG4G=tTTWJwRsGK4zNL2QHZ5+JJ8Jd-4yCLFkwxM4gLKNziw@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<CAMYG4G=tTTWJwRsGK4zNL2QHZ5+JJ8Jd-4yCLFkwxM4gLKNziw@mail.gmail.com>
Message-ID: <87va1z8eoe.fsf@jedbrown.org>

Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> writes:

>> 2) I tried all the suggestions mentioned before: setting
>> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
>> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
>> iterations of richardson/sor.
>>
>
> 1) Can we see some convergence for these? LU HAS to be better than the
> alternative, but you are saying that convergence still
> stalls above that I think. Okay, but I do not understand how SOR was
> working on the smaller problem, and cannot solve the coarse
> problem. However, it could be if the interpolation is crap, which it sounds
> like is true here.
>
> Jed, does that makes sense? If so, I would think about doing some bootstrap
> stuff.

Justin, can you compute and plot the spectrum of a small problem (<1000
dofs) as well as the eigenvectors of the extremal eigenvalues?

From chen2018 at purdue.edu  Mon Feb  4 15:17:07 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Mon, 4 Feb 2019 21:17:07 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
Message-ID: <DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>

Hi Mark,

     Will the parameter MatMPIAIJSetPreallocation in influence the following part

      do i=mystart,nelem,nproc
            call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
            ne=size(idx)
            idx=idx-1           !-1 since PETSC index starts from zero
            if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
            call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
      end do

I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:

 call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)


Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).

[0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.

[0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.

[0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used

[0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659

[0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70

[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.

[0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528

[0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528

[0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528

[0] VecScatterCreate_Seq(): Special case: sequential copy

[0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used

[0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0

[0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0

[0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.

 aseembel time   282.54871800000001

[0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256

[0] PCSetUp(): Setting up PC for first time

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged

[0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged


However, when I use the parallel mode,I got the following information:

[0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.

[0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.


it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)

Is there anything I am doing wrong?

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Mark Adams <mfadams at lbl.gov>
Sent: Tuesday, January 29, 2019 10:02 PM
To: Yaxiong Chen; PETSc users list
Subject: Re: [petsc-users] PETSC matrix assembling super slow

Optimized is a configuration flag not a versions.

You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.

You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.

On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu<mailto:chen2018 at purdue.edu>> wrote:

Thanks Mark,

    I use PETSC 3.9.4, is this the optimized version you called?

    Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: Tuesday, January 29, 2019 8:25 PM
To: Yaxiong Chen
Cc: Song Gao; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] PETSC matrix assembling super slow

Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.

You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).

I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.

And I trust you are running an optimized version of PETSc.


On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

Hi Song,

I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(Mat<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat> B,PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> d_nz,const PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> d_nnz[],PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> o_nz,const PetscInt<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt> o_nnz[])the same as my elemental matrix?

[cid:1689caf2cd3f456b1e51]

Thanks


Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907





________________________________
From: Song Gao <song.gao2 at mail.mcgill.ca<mailto:song.gao2 at mail.mcgill.ca>>
Sent: Tuesday, January 29, 2019 1:22 PM
To: Yaxiong Chen
Cc: Matthew Knepley; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] PETSC matrix assembling super slow

I think you would prefer to preallocate the matrix

uncomment this line
! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)



Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> a ?crit :

Hello,


I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?


Thanks


block
    PetscErrorCode ierr
     PetscMPIInt rank,nproc, mystart
     PetscInt  nelem
     integer,allocatable  ::info(:)
     real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
     integer, allocatable      :: idx(:)
     PetscScalar, pointer :: xx_v(:)

     PC               prec
     PetscScalar ::   val
     Vec              xvec,bvec,uvec
     Mat              Amat
     KSP              ksp
     PetscViewer viewer
     PetscInt  geq,j,k,ne,M,Istart,Iend
     PetscBool  flg
     KSPConvergedReason reason
     Vec         dummyVec, dummyVecs(1)
     MatNullSpace nullspace

     call PetscInitialize( PETSC_NULL_CHARACTER, ierr )

     if (ierr .ne. 0) then
     print*,'Unable to initialize PETSc'
     stop
     endif
     call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
     call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
     mystart=rank+1

 !   Parameter set
     info=ptsystem%getInitInfo()
     nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
     print*,'nelem',nelem
 !-------------------------------------
 !  Create Matrix
     call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
     call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
     call MatSetType( Amat, MATMPIBAIJ, ierr )
     ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
     call MatSetFromOptions( Amat, ierr )
     call MatSetUp( Amat, ierr )
     call MatGetOwnershipRange( Amat, Istart, Iend, ierr )

      xvec = tVec(0)
      call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
      call VecSetFromOptions( xvec, ierr )
      call VecDuplicate( xvec, bvec, ierr )
      call VecDuplicate( xvec, uvec, ierr )

   t1 = MPI_WTime();

      do i=mystart,nelem,nproc
            call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
            ne=size(idx)
            if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
             call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
      end do

      nelem   = info(Info_EleRHSNum)
      mystart = rank+1

      do i = mystart, nelem, nproc
        call ptSystem%getElementalRHS(i, RHSe, idx)
        print*,'idx',idx
        ne=size(idx)
        if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
      end do
      call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
      call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
      call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
     t2 = MPI_WTime();
      print*,'assembeling time',t2-t1
      ! Solve
      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.
      call KSPSetOperators(ksp,Amat,Amat,ierr)
      call KSPSetFromOptions(ksp,ierr)
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    call KSPSetType(ksp,KSPCG,ierr)
    call KSPGetPC(ksp,prec,ierr)
   ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
    call PCSetType(prec,PCJACOBI,ierr)
     call KSPSolve(ksp,bvec,xvec,ierr)
     call PetscFinalize(ierr)

end block



Yaxiong Chen,
Ph.D. Student


School of Mechanical Engineering, 3171

585 Purdue Mall

West Lafayette, IN 47907



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/bd946472/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3840 bytes
Desc: pastedImage.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190204/bd946472/attachment-0001.png>

From bsmith at mcs.anl.gov  Mon Feb  4 16:21:30 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 4 Feb 2019 22:21:30 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>



> On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi Mark,
>      Will the parameter MatMPIAIJSetPreallocation in influence the following part
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             idx=idx-1           !-1 since PETSC index starts from zero 
>             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
>             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
>       end do
> I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
>  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> 
> Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659


   Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?


> [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> [0] VecScatterCreate_Seq(): Special case: sequential copy
> [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
>  aseembel time   282.54871800000001     
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> [0] PCSetUp(): Setting up PC for first time
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> 
> However, when I use the parallel mode,I got the following information:
> [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.

   You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.

  Barry



> [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> 
> it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> Is there anything I am doing wrong?
> Thanks
> 
> Yaxiong Chen, 
> Ph.D. Student 
> 
> School of Mechanical Engineering, 3171 
> 585 Purdue Mall
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> From: Mark Adams <mfadams at lbl.gov>
> Sent: Tuesday, January 29, 2019 10:02 PM
> To: Yaxiong Chen; PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> Optimized is a configuration flag not a versions.
> 
> You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> 
> You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> 
> On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> Thanks Mark, 
>     I use PETSC 3.9.4, is this the optimized version you called?
>     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> 
> Yaxiong Chen, 
> Ph.D. Student 
> 
> School of Mechanical Engineering, 3171 
> 585 Purdue Mall
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> From: Mark Adams <mfadams at lbl.gov>
> Sent: Tuesday, January 29, 2019 8:25 PM
> To: Yaxiong Chen
> Cc: Song Gao; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> 
> You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> 
> I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> 
> And I trust you are running an optimized version of PETSc.
> 
> 
> On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi Song,
> I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix? 
> <pastedImage.png>
> 
> 
> Thanks
> 
> Yaxiong Chen, 
> Ph.D. Student 
> 
> School of Mechanical Engineering, 3171 
> 585 Purdue Mall
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> From: Song Gao <song.gao2 at mail.mcgill.ca>
> Sent: Tuesday, January 29, 2019 1:22 PM
> To: Yaxiong Chen
> Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> I think you would prefer to preallocate the matrix
> 
> uncomment this line 
> ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> 
> 
> 
> Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> Hello,
> 
> I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> 
> Thanks
> 
> block
>     PetscErrorCode ierr
>      PetscMPIInt rank,nproc, mystart
>      PetscInt  nelem
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
>      integer, allocatable      :: idx(:)
>      PetscScalar, pointer :: xx_v(:)
>   
>      PC               prec
>      PetscScalar ::   val
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>      PetscViewer viewer
>      PetscInt  geq,j,k,ne,M,Istart,Iend
>      PetscBool  flg
>      KSPConvergedReason reason
>      Vec         dummyVec, dummyVecs(1)
>      MatNullSpace nullspace
> 
>      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> 
>      if (ierr .ne. 0) then
>      print*,'Unable to initialize PETSc'
>      stop
>      endif
>      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
>      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>      mystart=rank+1
> 
>  !   Parameter set
>      info=ptsystem%getInitInfo()
>      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain) 
>      print*,'nelem',nelem
>  !-------------------------------------
>  !  Create Matrix
>      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
>      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
>      call MatSetType( Amat, MATMPIBAIJ, ierr )
>      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
>      call MatSetFromOptions( Amat, ierr )
>      call MatSetUp( Amat, ierr )
>      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
>  
>       xvec = tVec(0)
>       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
>       call VecSetFromOptions( xvec, ierr )
>       call VecDuplicate( xvec, bvec, ierr )
>       call VecDuplicate( xvec, uvec, ierr )
> 
>    t1 = MPI_WTime(); 
> 
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
>              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
>       end do
> 
>       nelem   = info(Info_EleRHSNum)
>       mystart = rank+1
> 
>       do i = mystart, nelem, nproc  
>         call ptSystem%getElementalRHS(i, RHSe, idx)
>         print*,'idx',idx
>         ne=size(idx)
>         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
>       end do 
>       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
>       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
>       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
>      t2 = MPI_WTime(); 
>       print*,'assembeling time',t2-t1
>       ! Solve
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>       call KSPSetOperators(ksp,Amat,Amat,ierr)
>       call KSPSetFromOptions(ksp,ierr)
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     call KSPSetType(ksp,KSPCG,ierr)
>     call KSPGetPC(ksp,prec,ierr)
>    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
>     call PCSetType(prec,PCJACOBI,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>      call PetscFinalize(ierr)
> 
> end block
> 
> 
> Yaxiong Chen, 
> Ph.D. Student 
> 
> School of Mechanical Engineering, 3171 
> 585 Purdue Mall
> West Lafayette, IN 47907


From bsmith at mcs.anl.gov  Mon Feb  4 16:37:13 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 4 Feb 2019 22:37:13 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
Message-ID: <A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>

> 
> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
> 
> Shri

  Shri

     How do you get access to this initial time-step later in the run? TSSetTimeStep() sets the value ts->time_step but I don't see any place where that initial timestep is stashed so could be used later.

   Thanks

    Barry


> On Feb 4, 2019, at 1:56 AM, Huck, Moritz via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi,
> yes there is one.
> My system has a set of very different time contants.
> At some events the fast one may be fully developped and only the slower ones are of concern after the event (depending on the postevent).
> If the last step is not an option, setting it manually might be helpfull since I generall know which timeconstants will be relevant (TSSetTime step seems not to have an effect during the postevent call).
> 
> Moritz
> ________________________________________
> Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
> Gesendet: Montag, 4. Februar 2019 05:09:24
> An: Huck, Moritz; Zhang, Hong
> Betreff: RE: [petsc-users] [TimeStepping] Eventhandler
> 
> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
> 
> Shri
> 
> -----Original Message-----
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Moritz Huck via petsc-users
> Sent: Sunday, February 3, 2019 10:45 AM
> To: Zhang, Hong <hongzhang at anl.gov>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
> 
> Hi,
> 
> I see I am sorry, I misinterpreted the output -ts_adapt_monitor.
> 
> You are right point 2 is not an issue.
> 
> I there a way to let the time stepping continue with the last time step instead of the initial time step?
> 
> Best Regards,
> 
> Moritz
> 
> Am 03.02.19 um 17:04 schrieb Zhang, Hong:
>> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>> 
>> 8 TS dt 0.5 time 3.11
>> 9 TS dt 0.5 time 3.61
>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>> 9 TS dt 0.467955 time 3.61 (r)
>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>> [3.61 - 4.07796]
>> 10 TS dt 0.00365179 time 4.07796
>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>> [4.07796 - 4.08161]
>> 11 TS dt 2.5403e-05 time 4.08161
>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 iterations
>> Ball hit the ground at t =  4.08 seconds
>> 12 TS dt 0.0283673 time 4.08163
>> 13 TS dt 0.01 time 4.11
>> 14 TS dt 0.1 time 4.12
>> 
>> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>> 
>> Hong (Mr.)
>> 
>>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Hi Barry,
>>> 
>>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>> 
>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>> [3.61 - 4.07796]
>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>> [4.07796 - 4.08161]
>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>>> iterations Ball hit the ground at t =  4.08 seconds
>>>       TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>> 
>>>       TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>> 
>>> It continues at the end of the "event interval" with the initial timestep.
>>> 
>>> Best Regards,
>>> 
>>> Moritz
>>> 
>>> 
>>> 
>>> 
>>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> Hi,
>>>>> I am trying to use SetEventHandler.
>>>>> I hav eencountered two "problems":
>>>>> 
>>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>> 
>>>>> 
>>>>> 2. the integration does not continue at the event encounter e.g. my
>>>>> eventfunction is g(t,u)=t-1, the event is found in the step from t=
>>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is 1.1
>>>>> + dt I would expect the next step to be 1.+dt
>>>>      Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>> 
>>>>    Thanks
>>>> 
>>>>    Barry
>>>> 
>>>>> Best Regards,
>>>>> Moritz
>>>>> 


From bsmith at mcs.anl.gov  Mon Feb  4 21:42:21 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 5 Feb 2019 03:42:21 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>



> On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> 
>   Hi Barry,
> 
>  !===================================================
>       mystart =rank +1                                    ! rank starts from 0
>       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process 
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
>             ne=size(idx)            
>             idx=idx-1           !-1 since PETSC index starts from zero 
>             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)  
>             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
>       end do
> !===================================================
>   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.

   This is very bad strategy because there is no data locality. 

> Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?

  What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.

   Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.

   Barry

> 
> Thanks
> 
> Yaxiong
> 
> 
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Monday, February 4, 2019 5:21 PM
> To: Yaxiong Chen
> Cc: Mark Adams; PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> 
> 
> > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hi Mark,
> >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> >       do i=mystart,nelem,nproc
> >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> >             ne=size(idx)
> >             idx=idx-1           !-1 since PETSC index starts from zero 
> >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> >       end do
> > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > 
> > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> 
> 
>    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> 
> 
> > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> >  aseembel time   282.54871800000001     
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > [0] PCSetUp(): Setting up PC for first time
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > 
> > However, when I use the parallel mode,I got the following information:
> > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> 
>    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> 
>   Barry
> 
> 
> 
> > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > 
> > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > Is there anything I am doing wrong?
> > Thanks
> > 
> > Yaxiong Chen, 
> > Ph.D. Student 
> > 
> > School of Mechanical Engineering, 3171 
> > 585 Purdue Mall
> > West Lafayette, IN 47907
> > 
> > 
> > 
> > 
> > 
> > From: Mark Adams <mfadams at lbl.gov>
> > Sent: Tuesday, January 29, 2019 10:02 PM
> > To: Yaxiong Chen; PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > Optimized is a configuration flag not a versions.
> > 
> > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > 
> > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > 
> > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > Thanks Mark, 
> >     I use PETSC 3.9.4, is this the optimized version you called?
> >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > 
> > Yaxiong Chen, 
> > Ph.D. Student 
> > 
> > School of Mechanical Engineering, 3171 
> > 585 Purdue Mall
> > West Lafayette, IN 47907
> > 
> > 
> > 
> > 
> > 
> > From: Mark Adams <mfadams at lbl.gov>
> > Sent: Tuesday, January 29, 2019 8:25 PM
> > To: Yaxiong Chen
> > Cc: Song Gao; petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > 
> > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > 
> > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > 
> > And I trust you are running an optimized version of PETSc.
> > 
> > 
> > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > Hi Song,
> > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix? 
> > <pastedImage.png>
> > 
> > 
> > Thanks
> > 
> > Yaxiong Chen, 
> > Ph.D. Student 
> > 
> > School of Mechanical Engineering, 3171 
> > 585 Purdue Mall
> > West Lafayette, IN 47907
> > 
> > 
> > 
> > 
> > 
> > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > Sent: Tuesday, January 29, 2019 1:22 PM
> > To: Yaxiong Chen
> > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > I think you would prefer to preallocate the matrix
> > 
> > uncomment this line 
> > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > 
> > 
> > 
> > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > Hello,
> > 
> > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > 
> > Thanks
> > 
> > block
> >     PetscErrorCode ierr
> >      PetscMPIInt rank,nproc, mystart
> >      PetscInt  nelem
> >      integer,allocatable  ::info(:)
> >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> >      integer, allocatable      :: idx(:)
> >      PetscScalar, pointer :: xx_v(:)
> >   
> >      PC               prec
> >      PetscScalar ::   val
> >      Vec              xvec,bvec,uvec
> >      Mat              Amat
> >      KSP              ksp
> >      PetscViewer viewer
> >      PetscInt  geq,j,k,ne,M,Istart,Iend
> >      PetscBool  flg
> >      KSPConvergedReason reason
> >      Vec         dummyVec, dummyVecs(1)
> >      MatNullSpace nullspace
> > 
> >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > 
> >      if (ierr .ne. 0) then
> >      print*,'Unable to initialize PETSc'
> >      stop
> >      endif
> >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> >      mystart=rank+1
> > 
> >  !   Parameter set
> >      info=ptsystem%getInitInfo()
> >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain) 
> >      print*,'nelem',nelem
> >  !-------------------------------------
> >  !  Create Matrix
> >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> >      call MatSetFromOptions( Amat, ierr )
> >      call MatSetUp( Amat, ierr )
> >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> >  
> >       xvec = tVec(0)
> >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> >       call VecSetFromOptions( xvec, ierr )
> >       call VecDuplicate( xvec, bvec, ierr )
> >       call VecDuplicate( xvec, uvec, ierr )
> > 
> >    t1 = MPI_WTime(); 
> > 
> >       do i=mystart,nelem,nproc
> >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> >             ne=size(idx)
> >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> >       end do
> > 
> >       nelem   = info(Info_EleRHSNum)
> >       mystart = rank+1
> > 
> >       do i = mystart, nelem, nproc  
> >         call ptSystem%getElementalRHS(i, RHSe, idx)
> >         print*,'idx',idx
> >         ne=size(idx)
> >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> >       end do 
> >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> >      t2 = MPI_WTime(); 
> >       print*,'assembeling time',t2-t1
> >       ! Solve
> >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > !  Set operators. Here the matrix that defines the linear system
> > !  also serves as the preconditioning matrix.
> >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> >       call KSPSetFromOptions(ksp,ierr)
> > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > !                      Solve the linear system
> > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >     call KSPSetType(ksp,KSPCG,ierr)
> >     call KSPGetPC(ksp,prec,ierr)
> >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> >     call PCSetType(prec,PCJACOBI,ierr)
> >      call KSPSolve(ksp,bvec,xvec,ierr)
> >      call PetscFinalize(ierr)
> > 
> > end block
> > 
> > 
> > Yaxiong Chen, 
> > Ph.D. Student 
> > 
> > School of Mechanical Engineering, 3171 
> > 585 Purdue Mall
> > West Lafayette, IN 47907


From bsmith at mcs.anl.gov  Mon Feb  4 21:53:25 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 5 Feb 2019 03:53:25 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
Message-ID: <9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>


  Ahh, thanks, for some reason I didn't find that.

  Based on Moritz's code example perhaps we need to add one or more TSEvent methods. The simplest thing would be to have a routine to allow reseting the "base" timestep that is used 1) within the TSEvent search and 2) after an event is found. Something like

   TSEventSetBaseTimeStep()

of course Base is a terrible name but I'm not sure what else to use.

   This is, of course, not a perfect solution because the user would have to know what value to use (though they could set it using values from TSGetTimeStep().)

   As an alternative or additional feature there could be a flag so that TSEvent proceeds with the current time step for 1) and 2) not the timestep_orig?

   It does seem absurd for the code to revert to a tiny step in dealing with the Event if the time stepper is happy with a much much large time-step for solving the ODE at that time range.

  Barry



   

> On Feb 4, 2019, at 7:48 PM, Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov> wrote:
> 
> It's stashed in event->timestep_orig. At the beginning of the TSEventHandler there is this code that sets timestep_orig.
> 
> if (event->status == TSEVENT_NONE) {
>    if (ts->steps == 1) /* After first successful step */
>      event->timestep_orig = ts->ptime - ts->ptime_prev;
> 
> After the step is resynchronized after event location, the next step chosen is the min. of the time-step before the event interval (event->timestep_prev) and the original step.
> 
> if (event->status == TSEVENT_RESET_NEXTSTEP) {
>    /* Restore time step */
>    dt = PetscMin(event->timestep_orig,event->timestep_prev);
>    ierr = TSSetTimeStep(ts,dt);CHKERRQ(ierr);
> 
> So, if the event->timestep_prev is smaller than event->timestep_orig (which could be possibly with adaptivity ON, it is allowed)
> 
> Shri
> 
> -----Original Message-----
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Smith, Barry F. via petsc-users
> Sent: Monday, February 4, 2019 4:37 PM
> To: Huck, Moritz <Moritz.Huck at rwth-aachen.de>
> Cc: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
> 
>> 
>> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
>> 
>> Shri
> 
>  Shri
> 
>     How do you get access to this initial time-step later in the run? TSSetTimeStep() sets the value ts->time_step but I don't see any place where that initial timestep is stashed so could be used later.
> 
>   Thanks
> 
>    Barry
> 
> 
>> On Feb 4, 2019, at 1:56 AM, Huck, Moritz via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> Hi,
>> yes there is one.
>> My system has a set of very different time contants.
>> At some events the fast one may be fully developped and only the slower ones are of concern after the event (depending on the postevent).
>> If the last step is not an option, setting it manually might be helpfull since I generall know which timeconstants will be relevant (TSSetTime step seems not to have an effect during the postevent call).
>> 
>> Moritz
>> ________________________________________
>> Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
>> Gesendet: Montag, 4. Februar 2019 05:09:24
>> An: Huck, Moritz; Zhang, Hong
>> Betreff: RE: [petsc-users] [TimeStepping] Eventhandler
>> 
>> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
>> 
>> Shri
>> 
>> -----Original Message-----
>> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of 
>> Moritz Huck via petsc-users
>> Sent: Sunday, February 3, 2019 10:45 AM
>> To: Zhang, Hong <hongzhang at anl.gov>
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>> 
>> Hi,
>> 
>> I see I am sorry, I misinterpreted the output -ts_adapt_monitor.
>> 
>> You are right point 2 is not an issue.
>> 
>> I there a way to let the time stepping continue with the last time step instead of the initial time step?
>> 
>> Best Regards,
>> 
>> Moritz
>> 
>> Am 03.02.19 um 17:04 schrieb Zhang, Hong:
>>> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>>> 
>>> 8 TS dt 0.5 time 3.11
>>> 9 TS dt 0.5 time 3.61
>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>> 9 TS dt 0.467955 time 3.61 (r)
>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>> [3.61 - 4.07796]
>>> 10 TS dt 0.00365179 time 4.07796
>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>> [4.07796 - 4.08161]
>>> 11 TS dt 2.5403e-05 time 4.08161
>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 
>>> iterations Ball hit the ground at t =  4.08 seconds
>>> 12 TS dt 0.0283673 time 4.08163
>>> 13 TS dt 0.01 time 4.11
>>> 14 TS dt 0.1 time 4.12
>>> 
>>> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>>> 
>>> Hong (Mr.)
>>> 
>>>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> Hi Barry,
>>>> 
>>>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>>> 
>>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>>> [3.61 - 4.07796]
>>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>>> [4.07796 - 4.08161]
>>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3 
>>>> iterations Ball hit the ground at t =  4.08 seconds
>>>>      TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>> 
>>>>      TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>> 
>>>> It continues at the end of the "event interval" with the initial timestep.
>>>> 
>>>> Best Regards,
>>>> 
>>>> Moritz
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> I am trying to use SetEventHandler.
>>>>>> I hav eencountered two "problems":
>>>>>> 
>>>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>>> 
>>>>>> 
>>>>>> 2. the integration does not continue at the event encounter e.g. 
>>>>>> my eventfunction is g(t,u)=t-1, the event is found in the step 
>>>>>> from t=
>>>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is 1.1
>>>>>> + dt I would expect the next step to be 1.+dt
>>>>>     Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>>> 
>>>>>   Thanks
>>>>> 
>>>>>   Barry
>>>>> 
>>>>>> Best Regards,
>>>>>> Moritz
>>>>>> 
> 


From bsmith at mcs.anl.gov  Mon Feb  4 21:59:29 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 5 Feb 2019 03:59:29 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
Message-ID: <8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>



> On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thanks everyone for your suggestions/feedback. So a few things:
> 
> 1) When I examined larger distribution networks (~150k buses) my eigenvalue estimates from the chebyshev method get enormous indeed. See below:
> 
> [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1, nnz/row (ave)=6, np=1
> [0] PCGAMGFilterGraph(): 	 98.5638% nnz after filtering, with threshold 0., 6.01293 nnz ave. (N=320745)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00 min=4.639523e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
> [0] PCGAMGFilterGraph(): 	 99.9753% nnz after filtering, with threshold 0., 7.32435 nnz ave. (N=44797)
> [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00 min=9.687506e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active pes
> [0] PCGAMGFilterGraph(): 	 99.684% nnz after filtering, with threshold 0., 22.5607 nnz ave. (N=13043)
> [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00 min=6.150856e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
> [0] PCGAMGFilterGraph(): 	 93.859% nnz after filtering, with threshold 0., 79.5142 nnz ave. (N=2256)
> [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00 min=6.780909e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
> [0] PCGAMGFilterGraph(): 	 99.1729% nnz after filtering, with threshold 0., 206.379 nnz ave. (N=232)
> [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00 min=2.153627e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
> 
> 2) I tried all the suggestions mentioned before: setting -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more iterations of richardson/sor.
> 
> 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just solving LU on the whole system.
> 
> 3b)The problem is that I can't get it to even speedup across 2 MPI processes for my 150k bus case (~300k dofs) - and I already checked that this problem could in theory be parallelized by setting the -ksp_max_it to something low and observing that the KSPSolve time decreases as MPI concurrency increases. The potential speedup is countered by the fact that the algorithmic convergence rate blows up when either more MPI processes are added or when I tune the block_size/overlap parameters. 

   Yes, we just haven't gotten a scalable preconditioner yet for your matrix.

> 
> Which leads me to one of two conclusions/questions:
> 
> A) Is my 300 DOF problem still too small? Do I need to look at a problem with, say, 10 Million DOF or more to see that increasing the asm_block_size will have better performance (despite having more KSP iterations) than a single giant asm_block?

   No, if you had a scalable preconditioner you would see good speed up with 300k unknowns, using 10 million won't help.

> 
> B) Or is there something even more sinister about my system of equations in complex form?

  There is a technical term for matrices for which both GAMG and ASM do very poorly on: nasty :-)

  Have you tried using parallel LU (for example ./configure --download-superlu_dist). Direct LU is the last refuge of the unpreconditionable. How large will your matrices get? 

   Barry

> 
> Thanks,
> Justin
> 
> On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com> wrote:
> The coarse grid is getting set to SOR rather than LU.
> 
>    Matt
> 
> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
> I tried these options:
> 
> -ksp_view
> -ksp_monitor_true_residual
> -ksp_type gmres
> -ksp_max_it 50
> -pc_type gamg
> -mg_coarse_pc_type sor
> -mg_levels_1_ksp_type richardson
> -mg_levels_2_ksp_type richardson
> -mg_levels_3_ksp_type richardson
> -mg_levels_4_ksp_type richardson
> -mg_levels_1_pc_type sor
> -mg_levels_2_pc_type sor
> -mg_levels_3_pc_type sor
> -mg_levels_4_pc_type sor
> 
> And still have a non-converging solution:
> 
>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=5 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.   0.   0.  
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>         Complexity:    grid = 1.31821
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=17, cols=17
>         total: nonzeros=163, allocated nonzeros=163
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 12 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=100, cols=100
>         total: nonzeros=1240, allocated nonzeros=1240
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=537, cols=537
>         total: nonzeros=5291, allocated nonzeros=5291
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=1541, cols=1541
>         total: nonzeros=8039, allocated nonzeros=8039
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=8541, cols=8541
>         total: nonzeros=46299, allocated nonzeros=46299
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 5464 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=8541, cols=8541
>     total: nonzeros=46299, allocated nonzeros=46299
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 5464 nodes, limit used is 5
> 
> Am I doing this right? Did I miss anything?
> 
> 
> 
> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
> Hi Mark,
> 
> 1) So with these options:
> 
> -ksp_type gmres
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -ksp_converged_reason
> -pc_type bjacobi
> 
> This is what I get:
> 
>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> Linear solve converged due to CONVERGED_RTOL iterations 5
> 
> 2) With richardson/sor:
> 
> Okay, its looks like Richardson/SOR solves this just fine. You can use this as the smoother for GAMG instead
> of Cheby/Jacobi, and probably see better results on the larger problems.
> 
>   Matt
>  
> -ksp_type richardson
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type sor
> 
> This is what I get:
> 
>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> Linear solve converged due to CONVERGED_RTOL iterations 40
> 
> 3) And lastly with chebyshev/jacobi:
> 
> -ksp_type chebyshev
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type jacobi
> 
>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> Linear solve converged due to CONVERGED_RTOL iterations 67
> 
> Looks like neither of these ksp/pc combos are good? I also tried -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the GAMG info I was able to grep from the out-of-box params:
> 
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph(): 	 99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00 min=2.394056e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 5.21674 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00 min=6.783380e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 9.85289 nnz ave. (N=537)
> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00 min=5.974776e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 12.4 nnz ave. (N=100)
> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00 min=4.842076e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> 
> And the one with pc_gamg_agg_nsmooths 0
> 
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph(): 	 99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 	 99.7467% nnz after filtering, with threshold 0., 3.07398 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 3.02211 nnz ave. (N=814)
> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 3.00434 nnz ave. (N=461)
> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=290)
> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=197)
> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 2.98425 nnz ave. (N=127)
> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 2.97561 nnz ave. (N=82)
> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 2.9697 nnz ave. (N=66)
> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> 
> 
> Thanks,
> Justin
> 
> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> 
> 
> Both GAMG and ILU are nice and dandy for this,
> 
> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and converge it way down, say rtol = 1.e-12. See which one is better in the early iteration and pick it. It would be nice to check that it solves the problem ...
> 
> The residual drops about 5 orders in the first iteration and then flatlines. That is very bad. Check that the smoother can actually solve the problem.  
>  
> but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG.
> 
> First try -pc_gamg_nsmooths 0
> 
> You can run with -info and grep on GAMG to see some diagnostic output. Eigen estimates are fragile in practice and with this parameter and SOR there are no eigen estimates needed. The max eigen values for all levels should be between say 2 (or less) and say 3-4. Much higher is a sign of a problem.
>  
> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
> 
> I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
> 
> Thanks
> 
> Justin
> 
> PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
> 
> There is a write up in the manual that is tutorial like.
>  
> 
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:
> Here's IMHO the simplest explanation of the equations I'm trying to solve:
> 
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> 
> Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
> 
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> small examples.
> 
>   Thanks,
> 
>     Matt
>  
> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi all,
> 
> I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers. 
> 
> I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
> 
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
> 
> 1) We really need to see the (simplified) equations
> 
> 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
> 
>  Thanks,
> 
>     Matt
>  
> Justin
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From M.Tiberga at tudelft.nl  Tue Feb  5 03:31:10 2019
From: M.Tiberga at tudelft.nl (Marco Tiberga)
Date: Tue, 5 Feb 2019 09:31:10 +0000
Subject: [petsc-users] Doubt on how to copy a Mat into another (Fortran)
In-Reply-To: <CAMYG4GkvETKZV5siTM=uoUh=O4PxwoSMfFOhLORR1teDbDSbeA@mail.gmail.com>
References: <63b35018005c4a88824e903f6b3f47a9@tudelft.nl>
	<CAMYG4GkvETKZV5siTM=uoUh=O4PxwoSMfFOhLORR1teDbDSbeA@mail.gmail.com>
Message-ID: <6939b78f9aac441dbda86facd8d2dffb@tudelft.nl>

Dear Matt,

Thanks for your reply, now it?s clear to me also why matArray is not destroyed, contrarily to Amat.

Do you think that declaring explicitly matArray as a (Fortran) pointer (at line 113) and then using ?matArray(1) => Amat? would be equivalent? It would be certainly clearer from the Fortran point of view.

With kind regards,
Marco Tiberga

From: Matthew Knepley [mailto:knepley at gmail.com]
Sent: maandag 4 februari 2019 17:46
To: Marco Tiberga
Cc: petsc-users at mcs.anl.gov; Aldo Hennink - TNW; Danny Lathouwers - TNW
Subject: Re: [petsc-users] Doubt on how to copy a Mat into another (Fortran)

On Mon, Feb 4, 2019 at 11:16 AM Marco Tiberga via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear PETSc developers,

Since I am learning how to use MatCreateNest, I was looking at example ex73f90t.F90<https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex73f90t.F90.html> (I am using Fortran).

At line 297, the array of matrices to be passed to MatCreateNest is initialized with code lines as ?matArray(1) = Amat?.

That is a pointer. It does not create a replica in memory, as the calls below do. Fortran is not explicit about this distinction
which can lead to confusion.

  Thanks,

    Matt


I was surprised to find such a simple command, I thought a matrix copy should be done by a sequence like
? call MatDuplicate(Amat, MAT_SHARE_NONZERO_PATTERN,matArray(1),ierr); CHKERRA(ierr)
  call MatCopy(Amat,matArray(1), SAME_NONZERO_PATTERN,ierr); CHKERRA(ierr) ?

So, I was wondering: are the two ways completely equivalent, and therefore the second unnecessarily more complex? or is the latter more robust and therefore preferable?
Is there any difference in terms of performance?

Thanks a lot for the clarification.

Best regards,
Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/





--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/4f3e27e7/attachment.html>

From eda.oktay at metu.edu.tr  Tue Feb  5 03:44:26 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 5 Feb 2019 12:44:26 +0300
Subject: [petsc-users] Problem in Loading Matrix in Examples
Message-ID: <CA+vu_XjHSW+9UWVK_F3Dg7w66WkgKTPKDgTEZy7tCG5ykS48Fw@mail.gmail.com>

Hi,

I am new in PETSc and trying to run some examples in MAT file.

I cannot run the ones that need to load matrix from file. Here are the
examples:

For ex12 in MAT file:
args: -f0 ${wPETSC_DIR}/share/petsc/datafiles/matrices/ns-real-int32-float64
requires: double !complex !define(PETSC_USE_64BIT_INDICES)

When I type ./ex11 -f0
home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64, I
get the following error message:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: This example is for exactly two processes
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex11 on a arch-linux2-c-debug named bb91.wls.metu.edu.tr
by edaoktay Tue Feb  5 12:41:49 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 31 in
/home/edaoktay/petsc-3.10.3/src/mat/examples/tutorials/ex11.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -f0
home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=56
:
system msg for write_line failure : Bad file descriptor


For ex10 in MAT file:
suffix: seqaij
requires: datafilespath double !complex !define(PETSC_USE_64BIT_INDICES)
args: -f ${DATAFILESPATH}/matrices/small -a_mat_type seqaij

There is not a matrix called small, however during configuration, it should
be tested successfully and there is an output. Also, I get pretty much the
same error here too.

I believe there is a problem in requirements but I cannot find any solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/8ba09cbf/attachment.html>

From andy.john.parker at gmail.com  Tue Feb  5 03:48:51 2019
From: andy.john.parker at gmail.com (Andrew Parker)
Date: Tue, 5 Feb 2019 09:48:51 +0000
Subject: [petsc-users] Store type (Eigen::Vector2d) in a petsc vec
Message-ID: <CAHJ1_RMvaJbXMM=gUS6VxhUQ9tP1jFHWK_9F0mYCvLp4Dwd0Kg@mail.gmail.com>

Hi,

I have quite a bit of working code that uses eigen vector2d/3d (and some
corresponding matrix3d types) to allow for local linAlg.  I would like to
store these within a petsc vector. So for example to have a vector for all
cells of velocity-vectors in 2d held for each cell within an eigen
vector2d, within a Petsc Vec for all cells.  What would be the best way to
achieve this within petsc? I would like to make full use of petsc
capabilities including parallel (ghost updating, partitioning etc),
likewise those of eigen's for local linAlg on small fixed sized vectors, so
how best to achieve this to allow for both eco-systems?
Thanks,
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/0f5ef974/attachment.html>

From knepley at gmail.com  Tue Feb  5 04:25:49 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 Feb 2019 05:25:49 -0500
Subject: [petsc-users] Doubt on how to copy a Mat into another (Fortran)
In-Reply-To: <6939b78f9aac441dbda86facd8d2dffb@tudelft.nl>
References: <63b35018005c4a88824e903f6b3f47a9@tudelft.nl>
	<CAMYG4GkvETKZV5siTM=uoUh=O4PxwoSMfFOhLORR1teDbDSbeA@mail.gmail.com>
	<6939b78f9aac441dbda86facd8d2dffb@tudelft.nl>
Message-ID: <CAMYG4GmyO7kxa41cZ0OVp=fouwZWymbVhvG0RSqiZpg21=Wt0Q@mail.gmail.com>

On Tue, Feb 5, 2019 at 4:31 AM Marco Tiberga <M.Tiberga at tudelft.nl> wrote:

> Dear Matt,
>
>
>
> Thanks for your reply, now it?s clear to me also why matArray is not
> destroyed, contrarily to Amat.
>
>
>
> Do you think that declaring explicitly matArray as a (Fortran) pointer (at
> line 113) and then using ?matArray(1) => Amat? would be equivalent? It
> would be certainly clearer from the Fortran point of view.
>

Looks like it. I am not a Fortran expert.

  Thanks,

    Matt


>
>
> With kind regards,
>
> Marco Tiberga
>
>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* maandag 4 februari 2019 17:46
> *To:* Marco Tiberga
> *Cc:* petsc-users at mcs.anl.gov; Aldo Hennink - TNW; Danny Lathouwers - TNW
> *Subject:* Re: [petsc-users] Doubt on how to copy a Mat into another
> (Fortran)
>
>
>
> On Mon, Feb 4, 2019 at 11:16 AM Marco Tiberga via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Dear PETSc developers,
>
>
>
> Since I am learning how to use MatCreateNest, I was looking at example
> ex73f90t.F90
> <https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex73f90t.F90.html>
> (I am using Fortran).
>
>
>
> At line 297, the array of matrices to be passed to MatCreateNest is
> initialized with code lines as ?matArray(1) = Amat?.
>
>
>
> That is a pointer. It does not create a replica in memory, as the calls
> below do. Fortran is not explicit about this distinction
>
> which can lead to confusion.
>
>
>
>   Thanks,
>
>
>
>     Matt
>
>
>
>
>
> I was surprised to find such a simple command, I thought a matrix copy
> should be done by a sequence like
>
> ? call MatDuplicate(Amat, MAT_SHARE_NONZERO_PATTERN,matArray(1),ierr);
> CHKERRA(ierr)
>
>   call MatCopy(Amat,matArray(1), SAME_NONZERO_PATTERN,ierr); CHKERRA(ierr)
> ?
>
>
>
> So, I was wondering: are the two ways completely equivalent, and therefore
> the second unnecessarily more complex? or is the latter more robust and
> therefore preferable?
>
> Is there any difference in terms of performance?
>
>
>
> Thanks a lot for the clarification.
>
>
>
> Best regards,
>
> Marco Tiberga
>
> PhD candidate
>
> Delft University of Technology
>
> Faculty of Applied Sciences
>
> Radiation Science & Technology Department
>
> Mekelweg 15, 2629 JB Delft, The Netherlands
>
> E-Mail: *m.tiberga at tudelft.nl <m.tiberga at tudelft.nl>*
>
> Website: http://www.nera.rst.tudelft.nl/
>
>
>
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/fd9cf234/attachment-0001.html>

From knepley at gmail.com  Tue Feb  5 04:35:17 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 Feb 2019 05:35:17 -0500
Subject: [petsc-users] Problem in Loading Matrix in Examples
In-Reply-To: <CA+vu_XjHSW+9UWVK_F3Dg7w66WkgKTPKDgTEZy7tCG5ykS48Fw@mail.gmail.com>
References: <CA+vu_XjHSW+9UWVK_F3Dg7w66WkgKTPKDgTEZy7tCG5ykS48Fw@mail.gmail.com>
Message-ID: <CAMYG4Gmecp8yEAd0QcfnfBJ9Nz0RdNG-6DbO5iniQoDHUxFdOg@mail.gmail.com>

On Tue, Feb 5, 2019 at 4:44 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I am new in PETSc and trying to run some examples in MAT file.
>
> I cannot run the ones that need to load matrix from file. Here are the
> examples:
>
> For ex12 in MAT file:
> args: -f0
> ${wPETSC_DIR}/share/petsc/datafiles/matrices/ns-real-int32-float64
> requires: double !complex !define(PETSC_USE_64BIT_INDICES)
>
> When I type ./ex11 -f0
> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64, I
> get the following error message:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: This example is for exactly two processes
>

As it says above, the example is for 2 processes, but you are running it on
1.

   Matt


> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./ex11 on a arch-linux2-c-debug named bb91.wls.metu.edu.tr
> by edaoktay Tue Feb  5 12:41:49 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [0]PETSC ERROR: #1 main() line 31 in
> /home/edaoktay/petsc-3.10.3/src/mat/examples/tutorials/ex11.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f0
> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=56
> :
> system msg for write_line failure : Bad file descriptor
>
>
> For ex10 in MAT file:
> suffix: seqaij
> requires: datafilespath double !complex !define(PETSC_USE_64BIT_INDICES)
> args: -f ${DATAFILESPATH}/matrices/small -a_mat_type seqaij
>
> There is not a matrix called small, however during configuration, it
> should be tested successfully and there is an output. Also, I get pretty
> much the same error here too.
>
> I believe there is a problem in requirements but I cannot find any
> solution.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/a272d400/attachment.html>

From eda.oktay at metu.edu.tr  Tue Feb  5 04:45:44 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 5 Feb 2019 13:45:44 +0300
Subject: [petsc-users] Problem in Loading Matrix in Examples
In-Reply-To: <CAMYG4Gmecp8yEAd0QcfnfBJ9Nz0RdNG-6DbO5iniQoDHUxFdOg@mail.gmail.com>
References: <CA+vu_XjHSW+9UWVK_F3Dg7w66WkgKTPKDgTEZy7tCG5ykS48Fw@mail.gmail.com>
	<CAMYG4Gmecp8yEAd0QcfnfBJ9Nz0RdNG-6DbO5iniQoDHUxFdOg@mail.gmail.com>
Message-ID: <CA+vu_XiQvJmfX=Fuc3k4=h3E8RUn_JFKCGxjsPS3mrG306DmrQ@mail.gmail.com>

Dear Matt,

Thank you for answering. I didn't recognize that part. But I still can't
find a matrix called "small" and there are some examples testing with
"medium". Am I missing something?

Thanks,

Eda

Matthew Knepley <knepley at gmail.com>, 5 ?ub 2019 Sal, 13:35 tarihinde ?unu
yazd?:

> On Tue, Feb 5, 2019 at 4:44 AM Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> I am new in PETSc and trying to run some examples in MAT file.
>>
>> I cannot run the ones that need to load matrix from file. Here are the
>> examples:
>>
>> For ex12 in MAT file:
>> args: -f0
>> ${wPETSC_DIR}/share/petsc/datafiles/matrices/ns-real-int32-float64
>> requires: double !complex !define(PETSC_USE_64BIT_INDICES)
>>
>> When I type ./ex11 -f0
>> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64, I
>> get the following error message:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: This example is for exactly two processes
>>
>
> As it says above, the example is for 2 processes, but you are running it
> on 1.
>
>    Matt
>
>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> [0]PETSC ERROR: ./ex11 on a arch-linux2-c-debug named
>> bb91.wls.metu.edu.tr by edaoktay Tue Feb  5 12:41:49 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> [0]PETSC ERROR: #1 main() line 31 in
>> /home/edaoktay/petsc-3.10.3/src/mat/examples/tutorials/ex11.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -f0
>> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0
>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=56
>> :
>> system msg for write_line failure : Bad file descriptor
>>
>>
>> For ex10 in MAT file:
>> suffix: seqaij
>> requires: datafilespath double !complex !define(PETSC_USE_64BIT_INDICES)
>> args: -f ${DATAFILESPATH}/matrices/small -a_mat_type seqaij
>>
>> There is not a matrix called small, however during configuration, it
>> should be tested successfully and there is an output. Also, I get pretty
>> much the same error here too.
>>
>> I believe there is a problem in requirements but I cannot find any
>> solution.
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/b78379d0/attachment.html>

From knepley at gmail.com  Tue Feb  5 05:51:32 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 Feb 2019 06:51:32 -0500
Subject: [petsc-users] Problem in Loading Matrix in Examples
In-Reply-To: <CA+vu_XiQvJmfX=Fuc3k4=h3E8RUn_JFKCGxjsPS3mrG306DmrQ@mail.gmail.com>
References: <CA+vu_XjHSW+9UWVK_F3Dg7w66WkgKTPKDgTEZy7tCG5ykS48Fw@mail.gmail.com>
	<CAMYG4Gmecp8yEAd0QcfnfBJ9Nz0RdNG-6DbO5iniQoDHUxFdOg@mail.gmail.com>
	<CA+vu_XiQvJmfX=Fuc3k4=h3E8RUn_JFKCGxjsPS3mrG306DmrQ@mail.gmail.com>
Message-ID: <CAMYG4GmcQFS397S7goEwPf5YtFr-dDg7ZCZHtPJ=tMFjKLsvHg@mail.gmail.com>

On Tue, Feb 5, 2019 at 5:45 AM Eda Oktay <eda.oktay at metu.edu.tr> wrote:

> Dear Matt,
>
> Thank you for answering. I didn't recognize that part. But I still can't
> find a matrix called "small" and there are some examples testing with
> "medium". Am I missing something?
>

We don't distribute the test matrices. They are on the FTP site:
http://ftp.mcs.anl.gov/pub/petsc/datafiles/matrices/

   Matt


> Thanks,
>
> Eda
>
> Matthew Knepley <knepley at gmail.com>, 5 ?ub 2019 Sal, 13:35 tarihinde ?unu
> yazd?:
>
>> On Tue, Feb 5, 2019 at 4:44 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I am new in PETSc and trying to run some examples in MAT file.
>>>
>>> I cannot run the ones that need to load matrix from file. Here are the
>>> examples:
>>>
>>> For ex12 in MAT file:
>>> args: -f0
>>> ${wPETSC_DIR}/share/petsc/datafiles/matrices/ns-real-int32-float64
>>> requires: double !complex !define(PETSC_USE_64BIT_INDICES)
>>>
>>> When I type ./ex11 -f0
>>> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64, I
>>> get the following error message:
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: This example is for exactly two processes
>>>
>>
>> As it says above, the example is for 2 processes, but you are running it
>> on 1.
>>
>>    Matt
>>
>>
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>> [0]PETSC ERROR: ./ex11 on a arch-linux2-c-debug named
>>> bb91.wls.metu.edu.tr by edaoktay Tue Feb  5 12:41:49 2019
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>> --download-metis --download-parmetis --download-superlu_dist
>>> --download-slepc --download-mpich
>>> [0]PETSC ERROR: #1 main() line 31 in
>>> /home/edaoktay/petsc-3.10.3/src/mat/examples/tutorials/ex11.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -f0
>>> home/petsc-3.10.3/share/petsc/datafiles/matrices/ns-real-int32-float64
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0
>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=56
>>> :
>>> system msg for write_line failure : Bad file descriptor
>>>
>>>
>>> For ex10 in MAT file:
>>> suffix: seqaij
>>> requires: datafilespath double !complex !define(PETSC_USE_64BIT_INDICES)
>>> args: -f ${DATAFILESPATH}/matrices/small -a_mat_type seqaij
>>>
>>> There is not a matrix called small, however during configuration, it
>>> should be tested successfully and there is an output. Also, I get pretty
>>> much the same error here too.
>>>
>>> I believe there is a problem in requirements but I cannot find any
>>> solution.
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/781a10f9/attachment.html>

From Moritz.Huck at rwth-aachen.de  Tue Feb  5 06:19:11 2019
From: Moritz.Huck at rwth-aachen.de (Huck, Moritz)
Date: Tue, 5 Feb 2019 12:19:11 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>,
	<9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
Message-ID: <0d77472b87c54dd2a9040612a405888e@rwth-aachen.de>

@Shri 
The system is very stiff, but the stiffness is handled well by ARKIMEX.

I'am using PETSc 3.10. (I cannot use 3.10.3 at the moment due to compatibilty with a third library), no special options for TS
Writing an MWE might take some time, if it is necessary I will write something at the weekend.

@Barry
A third option for 2) would be to let the adapator start from the event time, this might be usefull if a user doesnt know a good time step after the event.

Best Regards
Moritz
________________________________________
Von: Smith, Barry F. <bsmith at mcs.anl.gov>
Gesendet: Dienstag, 5. Februar 2019 04:53
An: Abhyankar, Shrirang G
Cc: Huck, Moritz; PETSc users list
Betreff: Re: [petsc-users] [TimeStepping] Eventhandler

  Ahh, thanks, for some reason I didn't find that.

  Based on Moritz's code example perhaps we need to add one or more TSEvent methods. The simplest thing would be to have a routine to allow reseting the "base" timestep that is used 1) within the TSEvent search and 2) after an event is found. Something like

   TSEventSetBaseTimeStep()

of course Base is a terrible name but I'm not sure what else to use.

   This is, of course, not a perfect solution because the user would have to know what value to use (though they could set it using values from TSGetTimeStep().)

   As an alternative or additional feature there could be a flag so that TSEvent proceeds with the current time step for 1) and 2) not the timestep_orig?

   It does seem absurd for the code to revert to a tiny step in dealing with the Event if the time stepper is happy with a much much large time-step for solving the ODE at that time range.

  Barry





> On Feb 4, 2019, at 7:48 PM, Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov> wrote:
>
> It's stashed in event->timestep_orig. At the beginning of the TSEventHandler there is this code that sets timestep_orig.
>
> if (event->status == TSEVENT_NONE) {
>    if (ts->steps == 1) /* After first successful step */
>      event->timestep_orig = ts->ptime - ts->ptime_prev;
>
> After the step is resynchronized after event location, the next step chosen is the min. of the time-step before the event interval (event->timestep_prev) and the original step.
>
> if (event->status == TSEVENT_RESET_NEXTSTEP) {
>    /* Restore time step */
>    dt = PetscMin(event->timestep_orig,event->timestep_prev);
>    ierr = TSSetTimeStep(ts,dt);CHKERRQ(ierr);
>
> So, if the event->timestep_prev is smaller than event->timestep_orig (which could be possibly with adaptivity ON, it is allowed)
>
> Shri
>
> -----Original Message-----
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Smith, Barry F. via petsc-users
> Sent: Monday, February 4, 2019 4:37 PM
> To: Huck, Moritz <Moritz.Huck at rwth-aachen.de>
> Cc: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>
>>
>> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
>>
>> Shri
>
>  Shri
>
>     How do you get access to this initial time-step later in the run? TSSetTimeStep() sets the value ts->time_step but I don't see any place where that initial timestep is stashed so could be used later.
>
>   Thanks
>
>    Barry
>
>
>> On Feb 4, 2019, at 1:56 AM, Huck, Moritz via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hi,
>> yes there is one.
>> My system has a set of very different time contants.
>> At some events the fast one may be fully developped and only the slower ones are of concern after the event (depending on the postevent).
>> If the last step is not an option, setting it manually might be helpfull since I generall know which timeconstants will be relevant (TSSetTime step seems not to have an effect during the postevent call).
>>
>> Moritz
>> ________________________________________
>> Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
>> Gesendet: Montag, 4. Februar 2019 05:09:24
>> An: Huck, Moritz; Zhang, Hong
>> Betreff: RE: [petsc-users] [TimeStepping] Eventhandler
>>
>> Is there any reason you'd like to use the last time step? We used the initial time-step (which is assumed to be small) so as (a) to minimize any numerical issues that may be caused by a large last time-step, and (b) to not miss any ensuing events that may be triggered by the discontinuity.
>>
>> Shri
>>
>> -----Original Message-----
>> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of
>> Moritz Huck via petsc-users
>> Sent: Sunday, February 3, 2019 10:45 AM
>> To: Zhang, Hong <hongzhang at anl.gov>
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>>
>> Hi,
>>
>> I see I am sorry, I misinterpreted the output -ts_adapt_monitor.
>>
>> You are right point 2 is not an issue.
>>
>> I there a way to let the time stepping continue with the last time step instead of the initial time step?
>>
>> Best Regards,
>>
>> Moritz
>>
>> Am 03.02.19 um 17:04 schrieb Zhang, Hong:
>>> ex40 seems to work fine. Here is the output with -ts_monitor -ts_event_monitor -ts_dt 0.01 :
>>>
>>> 8 TS dt 0.5 time 3.11
>>> 9 TS dt 0.5 time 3.61
>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>> 9 TS dt 0.467955 time 3.61 (r)
>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>> [3.61 - 4.07796]
>>> 10 TS dt 0.00365179 time 4.07796
>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>> [4.07796 - 4.08161]
>>> 11 TS dt 2.5403e-05 time 4.08161
>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>>> iterations Ball hit the ground at t =  4.08 seconds
>>> 12 TS dt 0.0283673 time 4.08163
>>> 13 TS dt 0.01 time 4.11
>>> 14 TS dt 0.1 time 4.12
>>>
>>> The event occurs during the step [3.61- 4.11]. After the event is located at 4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see 12 TS) to complete the original step. After 4.11, the specified initial stepsize dt=0.01 will be used.
>>>
>>> Hong (Mr.)
>>>
>>>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>
>>>> Hi Barry,
>>>>
>>>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-2) has the same behavior:
>>>>
>>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>>> [3.61 - 4.07796]
>>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>>> [4.07796 - 4.08161]
>>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>>>> iterations Ball hit the ground at t =  4.08 seconds
>>>>      TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02 dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>>
>>>>      TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01 dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>>
>>>> It continues at the end of the "event interval" with the initial timestep.
>>>>
>>>> Best Regards,
>>>>
>>>> Moritz
>>>>
>>>>
>>>>
>>>>
>>>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I am trying to use SetEventHandler.
>>>>>> I hav eencountered two "problems":
>>>>>>
>>>>>> 1. after an event is encountered the next time step is needlessly small (h=1e-6), even if a don't change anything in the function (for testing).
>>>>>>
>>>>>>
>>>>>> 2. the integration does not continue at the event encounter e.g.
>>>>>> my eventfunction is g(t,u)=t-1, the event is found in the step
>>>>>> from t=
>>>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is 1.1
>>>>>> + dt I would expect the next step to be 1.+dt
>>>>>     Hmm, this sounds like it could be an error. Could you send us a simple code that exhibits it?
>>>>>
>>>>>   Thanks
>>>>>
>>>>>   Barry
>>>>>
>>>>>> Best Regards,
>>>>>> Moritz
>>>>>>
>


From huq2090 at gmail.com  Tue Feb  5 08:05:08 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Tue, 5 Feb 2019 08:05:08 -0600
Subject: [petsc-users] Installing PETSc
Message-ID: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>

Hello PETSc Developers,

may be this is a trivial question!

I usually run PETSc code from Home/petsc-3.10.2 directory. Last day I tried
to run the code from Documents/petsc directory but I can't. As far as I can
recall, I have installed PETSc in the Home directory. Is it the reason why
I can't run PETSc code from other directory? Shall I install PETSc in the
root directory?

Again, if I run command "which petsc" I don't get any echo on the terminal.

Thanks.
Sincerely,
Huq


-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/8cfff6fe/attachment.html>

From jed at jedbrown.org  Tue Feb  5 08:22:21 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 Feb 2019 07:22:21 -0700
Subject: [petsc-users] Store type (Eigen::Vector2d) in a petsc vec
In-Reply-To: <CAHJ1_RMvaJbXMM=gUS6VxhUQ9tP1jFHWK_9F0mYCvLp4Dwd0Kg@mail.gmail.com>
References: <CAHJ1_RMvaJbXMM=gUS6VxhUQ9tP1jFHWK_9F0mYCvLp4Dwd0Kg@mail.gmail.com>
Message-ID: <87ftt271sy.fsf@jedbrown.org>

My suggestion is to use PETSc like usual and inside your
residual/Jacobian evaluation, for each cell or batch of cells, create
Eigen objects.  For size 2d or 3d, it won't matter much whether you make
them share memory with the PETSc Vec -- the Eigen types should mostly
exist in registers.

Andrew Parker via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi,
>
> I have quite a bit of working code that uses eigen vector2d/3d (and some
> corresponding matrix3d types) to allow for local linAlg.  I would like to
> store these within a petsc vector. So for example to have a vector for all
> cells of velocity-vectors in 2d held for each cell within an eigen
> vector2d, within a Petsc Vec for all cells.  What would be the best way to
> achieve this within petsc? I would like to make full use of petsc
> capabilities including parallel (ghost updating, partitioning etc),
> likewise those of eigen's for local linAlg on small fixed sized vectors, so
> how best to achieve this to allow for both eco-systems?
> Thanks,
> Andy

From jed at jedbrown.org  Tue Feb  5 08:30:38 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 Feb 2019 07:30:38 -0700
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <0d77472b87c54dd2a9040612a405888e@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
	<0d77472b87c54dd2a9040612a405888e@rwth-aachen.de>
Message-ID: <87d0o671f5.fsf@jedbrown.org>

"Huck, Moritz via petsc-users" <petsc-users at mcs.anl.gov> writes:

> @Shri 
> The system is very stiff, but the stiffness is handled well by ARKIMEX.
>
> I'am using PETSc 3.10. (I cannot use 3.10.3 at the moment due to
> compatibilty with a third library), 

What compatibility problem is this?  3.10.3 should be (binary and
source) backward compatible with other 3.10 releases, so if there is a
case where it is not, we need to understand it.


Regarding the event, it seems there are scenarios where something
important changes in the model and a conservative time step is
desirable, and other scenarios where the dynamics after the event are on
the same scale as before so it makes sense to continue with the same
step sizes.

From balay at mcs.anl.gov  Tue Feb  5 08:35:41 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Tue, 5 Feb 2019 14:35:41 +0000
Subject: [petsc-users] Installing PETSc
In-Reply-To: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>
References: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1902050831590.20719@sb>

On Tue, 5 Feb 2019, Fazlul Huq via petsc-users wrote:

> Hello PETSc Developers,
> 
> may be this is a trivial question!
> 
> I usually run PETSc code from Home/petsc-3.10.2 directory. Last day I tried
> to run the code from Documents/petsc directory but I can't.

I don't know what you mean here - please copy/paste your attempt - and errors you've got.

> As far as I can
> recall, I have installed PETSc in the Home directory. Is it the reason why
> I can't run PETSc code from other directory?

nope

> Shall I install PETSc in the root directory?

Once you have a working install of petsc library - you should be able to use it from anywhere
in the file-system.

> 
> Again, if I run command "which petsc" I don't get any echo on the terminal.

There is no 'petsc' binary in petsc library.

Satish

From jed at jedbrown.org  Tue Feb  5 08:36:32 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 Feb 2019 07:36:32 -0700
Subject: [petsc-users] Installing PETSc
In-Reply-To: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>
References: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>
Message-ID: <87a7ja715b.fsf@jedbrown.org>

Fazlul Huq via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hello PETSc Developers,
>
> may be this is a trivial question!
>
> I usually run PETSc code from Home/petsc-3.10.2 directory. Last day I tried
> to run the code from Documents/petsc directory but I can't. As far as I can
> recall, I have installed PETSc in the Home directory. Is it the reason why
> I can't run PETSc code from other directory? Shall I install PETSc in the
> root directory?

What do you mean by "run PETSc"?  How did you link your executable?

> Again, if I run command "which petsc" I don't get any echo on the terminal.

That is searching for an executable named "petsc" in your PATH.

  echo $PATH

to see the value of that variable.

From andy.john.parker at gmail.com  Tue Feb  5 08:43:01 2019
From: andy.john.parker at gmail.com (Andrew Parker)
Date: Tue, 5 Feb 2019 14:43:01 +0000
Subject: [petsc-users] Store type (Eigen::Vector2d) in a petsc vec
In-Reply-To: <87ftt271sy.fsf@jedbrown.org>
References: <CAHJ1_RMvaJbXMM=gUS6VxhUQ9tP1jFHWK_9F0mYCvLp4Dwd0Kg@mail.gmail.com>
	<87ftt271sy.fsf@jedbrown.org>
Message-ID: <CAHJ1_RNYJo7SmfA7wAuLVZ37UZ=KAft0US2ELHEeeKD8eX4fWQ@mail.gmail.com>

Thanks, so you would suggest a flat vector storing u, v, w (or indeed x, y,
z) or interleaved and then construct eigen types on the fly?  Can I ask, is
that because Vec cannot store user defined types (as in it's not
templatetable?)

Thanks,
Andy

On Tue, 5 Feb 2019 at 14:22, Jed Brown <jed at jedbrown.org> wrote:

> My suggestion is to use PETSc like usual and inside your
> residual/Jacobian evaluation, for each cell or batch of cells, create
> Eigen objects.  For size 2d or 3d, it won't matter much whether you make
> them share memory with the PETSc Vec -- the Eigen types should mostly
> exist in registers.
>
> Andrew Parker via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Hi,
> >
> > I have quite a bit of working code that uses eigen vector2d/3d (and some
> > corresponding matrix3d types) to allow for local linAlg.  I would like to
> > store these within a petsc vector. So for example to have a vector for
> all
> > cells of velocity-vectors in 2d held for each cell within an eigen
> > vector2d, within a Petsc Vec for all cells.  What would be the best way
> to
> > achieve this within petsc? I would like to make full use of petsc
> > capabilities including parallel (ghost updating, partitioning etc),
> > likewise those of eigen's for local linAlg on small fixed sized vectors,
> so
> > how best to achieve this to allow for both eco-systems?
> > Thanks,
> > Andy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/9f864fe4/attachment.html>

From andy.john.parker at gmail.com  Tue Feb  5 08:46:14 2019
From: andy.john.parker at gmail.com (Andrew Parker)
Date: Tue, 5 Feb 2019 14:46:14 +0000
Subject: [petsc-users] MWE for DMPlexCreateCGNS
Message-ID: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>

Does anyone have a MWE for DMPlexCreateCGNS to use in parallel? Ideally,
read parallel, distribute in parallel, construct ghost cells (for parallel
comms and halos for physical boundaries)? It's for a cell-centered solver
working with cgns meshes.  Is there any limitation on cell-types?

Thanks,
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/ceb1d420/attachment.html>

From jed at jedbrown.org  Tue Feb  5 08:59:19 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 Feb 2019 07:59:19 -0700
Subject: [petsc-users] Store type (Eigen::Vector2d) in a petsc vec
In-Reply-To: <CAHJ1_RNYJo7SmfA7wAuLVZ37UZ=KAft0US2ELHEeeKD8eX4fWQ@mail.gmail.com>
References: <CAHJ1_RMvaJbXMM=gUS6VxhUQ9tP1jFHWK_9F0mYCvLp4Dwd0Kg@mail.gmail.com>
	<87ftt271sy.fsf@jedbrown.org>
	<CAHJ1_RNYJo7SmfA7wAuLVZ37UZ=KAft0US2ELHEeeKD8eX4fWQ@mail.gmail.com>
Message-ID: <877eee703c.fsf@jedbrown.org>

Andrew Parker <andy.john.parker at gmail.com> writes:

> Thanks, so you would suggest a flat vector storing u, v, w (or indeed x, y,
> z) or interleaved and then construct eigen types on the fly?  

Interleaved if you want to use Eigen types in the same memory, or if
your code (like most applications) benefits more from memory locality
than from vectorization.  You can cast because Matrix<double,2,1> has
the same storage representation as

struct {
  double values[2];
};

> Can I ask, is that because Vec cannot store user defined types (as in
> it's not templatetable?)

PETSc does not use templates, but you wouldn't want it to in this
circumstance.  Either your types have the same storage representation
and you can cast or they don't and templating would ruin what are now
contiguous operations.

From knepley at gmail.com  Tue Feb  5 09:27:21 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 Feb 2019 10:27:21 -0500
Subject: [petsc-users] MWE for DMPlexCreateCGNS
In-Reply-To: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>
References: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>
Message-ID: <CAMYG4GkocLUpbcfrKGywqxqHJUq4thARMV4c5BzV-9Xv2apfhQ@mail.gmail.com>

On Tue, Feb 5, 2019 at 9:47 AM Andrew Parker via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Does anyone have a MWE for DMPlexCreateCGNS to use in parallel? Ideally,
> read parallel, distribute in parallel, construct ghost cells (for parallel
> comms and halos for physical boundaries)? It's for a cell-centered solver
> working with cgns meshes.  Is there any limitation on cell-types?
>

1) CGNS is a terrible format, unfortunately. Are you sure about committing
to it? Always best to question design decisions before lots of code has
been written.

2) DMPlexCreateCGNS() probably works, but it has not been tested
exhaustively.

3) It does not work in parallel, and is unlikely to in the near future. In
order to read in parallel, you should be able to nicely select
    blocks from disk in the format, and also have local BC specification
(rather than global numbers). MED is a nice format like this.
    It is the CASCADE format, and GMsh uses it internally. We can now read
MED in parallel. Without too much work, we could also
    read ExodusII in parallel I think.

4) Once read, you get a regular Plex, so you can redistribute, make ghost
cells, etc. as you can for any Plex mesh.

5) As long as all you want Plex to do is manage topology and field data,
then you can have whatever mix of cell types you want.

6) Limitations on cell types come from routines that calculate geometric
quantities, integrate, etc.

  Thanks,

     Matt


> Thanks,
> Andy
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/06667680/attachment.html>

From andy.john.parker at gmail.com  Tue Feb  5 10:13:02 2019
From: andy.john.parker at gmail.com (Andrew Parker)
Date: Tue, 5 Feb 2019 16:13:02 +0000
Subject: [petsc-users] MWE for DMPlexCreateCGNS
In-Reply-To: <CAMYG4GkocLUpbcfrKGywqxqHJUq4thARMV4c5BzV-9Xv2apfhQ@mail.gmail.com>
References: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>
	<CAMYG4GkocLUpbcfrKGywqxqHJUq4thARMV4c5BzV-9Xv2apfhQ@mail.gmail.com>
Message-ID: <CAHJ1_RMd=kcYAV7DuXwgC8svKLHf_0cOYUGj7BP-8FjAYDpFOA@mail.gmail.com>

On Tue, 5 Feb 2019 at 15:27, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Feb 5, 2019 at 9:47 AM Andrew Parker via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Does anyone have a MWE for DMPlexCreateCGNS to use in parallel? Ideally,
>> read parallel, distribute in parallel, construct ghost cells (for parallel
>> comms and halos for physical boundaries)? It's for a cell-centered solver
>> working with cgns meshes.  Is there any limitation on cell-types?
>>
>
> 1) CGNS is a terrible format, unfortunately. Are you sure about committing
> to it? Always best to question design decisions before lots of code has
> been written.
>

> 2) DMPlexCreateCGNS() probably works, but it has not been tested
> exhaustively.
>

> 3) It does not work in parallel, and is unlikely to in the near future. In
> order to read in parallel, you should be able to nicely select
>     blocks from disk in the format, and also have local BC specification
> (rather than global numbers). MED is a nice format like this.
>     It is the CASCADE format, and GMsh uses it internally. We can now read
> MED in parallel. Without too much work, we could also
>     read ExodusII in parallel I think.
>

Do you have a MWE for either of those two formats then?

Key for me is to be in parallel by the time the setup stage is finished, if
there is some serial reading here and there at the initial phase that's
fine.  For a cell-centered code I just want to go from file read to
parallel distribute, parallel ghost-cell creation, physical halo creation,
and boundary condition (faces, edge in 2D) marking re-using as much of the
petsc DMPlex system as possible.

My main requirement with any file format is the ability in a mesh/cad
package to mark complex groups of edges/faces/cells together (on the
boundary but also groupings of cells in the volume) to infer groups and
therefore bcs, or material regions etc.  I am aware med/exo provide this:
my reason for picking cgns is that is central to their format also.  I need
to ensure for complex geometry and meshes, that if the file format does
encode groupings/bcs then these are accessible after the parallel
distribution phase in complete.  It would help me massively if there were
an example that does all of this....I'll jump ship on file format to get
that.

Thanks,
Andy

>
> 4) Once read, you get a regular Plex, so you can redistribute, make ghost
> cells, etc. as you can for any Plex mesh.
>
> 5) As long as all you want Plex to do is manage topology and field data,
> then you can have whatever mix of cell types you want.
>
> 6) Limitations on cell types come from routines that calculate geometric
> quantities, integrate, etc.
>
>   Thanks,
>
>      Matt
>


>
>
>> Thanks,
>> Andy
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/f4872b72/attachment.html>

From mfadams at lbl.gov  Tue Feb  5 10:26:37 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 5 Feb 2019 11:26:37 -0500
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
Message-ID: <CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>

I would stay away from eigen estimates in the solver (but give us the
spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.

Applications that have lived on direct solvers can add sorts of crap like
penalty terms.

sor seemed to work OK so I'd check the coarse grids in GAMG. Test with just
two levels. That way you don't have to use sor on an internal MG grid,
which can be bad and probably is here.


On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> > On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Thanks everyone for your suggestions/feedback. So a few things:
> >
> > 1) When I examined larger distribution networks (~150k buses) my
> eigenvalue estimates from the chebyshev method get enormous indeed. See
> below:
> >
> > [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
> nnz/row (ave)=6, np=1
> > [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with
> threshold 0., 6.01293 nnz ave. (N=320745)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
> min=4.639523e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with
> threshold 0., 7.32435 nnz ave. (N=44797)
> > [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
> min=9.687506e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1
> active pes
> > [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with
> threshold 0., 22.5607 nnz ave. (N=13043)
> > [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
> min=6.150856e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active
> pes
> > [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with
> threshold 0., 79.5142 nnz ave. (N=2256)
> > [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
> min=6.780909e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with
> threshold 0., 206.379 nnz ave. (N=232)
> > [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
> min=2.153627e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
> >
> > 2) I tried all the suggestions mentioned before: setting
> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
> iterations of richardson/sor.
> >
> > 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just
> solving LU on the whole system.
> >
> > 3b)The problem is that I can't get it to even speedup across 2 MPI
> processes for my 150k bus case (~300k dofs) - and I already checked that
> this problem could in theory be parallelized by setting the -ksp_max_it to
> something low and observing that the KSPSolve time decreases as MPI
> concurrency increases. The potential speedup is countered by the fact that
> the algorithmic convergence rate blows up when either more MPI processes
> are added or when I tune the block_size/overlap parameters.
>
>    Yes, we just haven't gotten a scalable preconditioner yet for your
> matrix.
>
> >
> > Which leads me to one of two conclusions/questions:
> >
> > A) Is my 300 DOF problem still too small? Do I need to look at a problem
> with, say, 10 Million DOF or more to see that increasing the asm_block_size
> will have better performance (despite having more KSP iterations) than a
> single giant asm_block?
>
>    No, if you had a scalable preconditioner you would see good speed up
> with 300k unknowns, using 10 million won't help.
>
> >
> > B) Or is there something even more sinister about my system of equations
> in complex form?
>
>   There is a technical term for matrices for which both GAMG and ASM do
> very poorly on: nasty :-)
>
>   Have you tried using parallel LU (for example ./configure
> --download-superlu_dist). Direct LU is the last refuge of the
> unpreconditionable. How large will your matrices get?
>
>    Barry
>
> >
> > Thanks,
> > Justin
> >
> > On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > The coarse grid is getting set to SOR rather than LU.
> >
> >    Matt
> >
> > On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
> > I tried these options:
> >
> > -ksp_view
> > -ksp_monitor_true_residual
> > -ksp_type gmres
> > -ksp_max_it 50
> > -pc_type gamg
> > -mg_coarse_pc_type sor
> > -mg_levels_1_ksp_type richardson
> > -mg_levels_2_ksp_type richardson
> > -mg_levels_3_ksp_type richardson
> > -mg_levels_4_ksp_type richardson
> > -mg_levels_1_pc_type sor
> > -mg_levels_2_pc_type sor
> > -mg_levels_3_pc_type sor
> > -mg_levels_4_pc_type sor
> >
> > And still have a non-converging solution:
> >
> >   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
> >   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
> >   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
> >   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
> >   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
> >   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
> >   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
> >   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
> >   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> >  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> >  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> >  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> >  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> >  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> >  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> >  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> >  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> >  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> >  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> >  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> >  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> >  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> >  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> >  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> >  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> >  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> >  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> >  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> >  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> >  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> >  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> >  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> >  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> >  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> >  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> >  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> >  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> >  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> >  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> >  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> >  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> >  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> >  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> >  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> >  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> >  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> >  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> >  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> >  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> >  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> > KSP Object: 1 MPI processes
> >   type: gmres
> >     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >     happy breakdown tolerance 1e-30
> >   maximum iterations=50, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: gamg
> >     type is MULTIPLICATIVE, levels=5 cycles=v
> >       Cycles per PCApply=1
> >       Using externally compute Galerkin coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values in graph on each level =
>  0.   0.   0.
> >         Threshold scaling factor for each level not specified = 1.
> >         AGG specific options
> >           Symmetric graph false
> >           Number of levels to square graph 1
> >           Number smoothing steps 1
> >         Complexity:    grid = 1.31821
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object: (mg_coarse_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_coarse_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=17, cols=17
> >         total: nonzeros=163, allocated nonzeros=163
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 12 nodes, limit used is 5
> >   Down solver (pre-smoother) on level 1 -------------------------------
> >     KSP Object: (mg_levels_1_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_1_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=100, cols=100
> >         total: nonzeros=1240, allocated nonzeros=1240
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 2 -------------------------------
> >     KSP Object: (mg_levels_2_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_2_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=537, cols=537
> >         total: nonzeros=5291, allocated nonzeros=5291
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 3 -------------------------------
> >     KSP Object: (mg_levels_3_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_3_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=1541, cols=1541
> >         total: nonzeros=8039, allocated nonzeros=8039
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 4 -------------------------------
> >     KSP Object: (mg_levels_4_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_4_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=8541, cols=8541
> >         total: nonzeros=46299, allocated nonzeros=46299
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 5464 nodes, limit used is 5
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> >     type: seqaij
> >     rows=8541, cols=8541
> >     total: nonzeros=46299, allocated nonzeros=46299
> >     total number of mallocs used during MatSetValues calls =0
> >       using I-node routines: found 5464 nodes, limit used is 5
> >
> > Am I doing this right? Did I miss anything?
> >
> >
> >
> > On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
> > Hi Mark,
> >
> > 1) So with these options:
> >
> > -ksp_type gmres
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -ksp_converged_reason
> > -pc_type bjacobi
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
> >   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
> >   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
> >   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
> >   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 5
> >
> > 2) With richardson/sor:
> >
> > Okay, its looks like Richardson/SOR solves this just fine. You can use
> this as the smoother for GAMG instead
> > of Cheby/Jacobi, and probably see better results on the larger problems.
> >
> >   Matt
> >
> > -ksp_type richardson
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type sor
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
> >   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
> >   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
> >   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
> >   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
> >   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
> >   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
> >   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
> >   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
> >  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
> >  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
> >  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
> >  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
> >  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
> >  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
> >  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
> >  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
> >  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
> >  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
> >  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
> >  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
> >  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
> >  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
> >  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
> >  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
> >  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
> >  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
> >  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
> >  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
> >  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
> >  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
> >  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
> >  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
> >  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
> >  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
> >  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
> >  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
> >  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
> >  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
> >  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 40
> >
> > 3) And lastly with chebyshev/jacobi:
> >
> > -ksp_type chebyshev
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type jacobi
> >
> >   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
> >   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
> >   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
> >   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
> >   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
> >   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
> >   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
> >   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
> >   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
> >   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
> >  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
> >  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
> >  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
> >  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
> >  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
> >  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
> >  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
> >  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
> >  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
> >  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
> >  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
> >  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
> >  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
> >  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
> >  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
> >  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
> >  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
> >  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
> >  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
> >  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
> >  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
> >  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
> >  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
> >  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
> >  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
> >  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
> >  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
> >  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
> >  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
> >  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
> >  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
> >  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
> >  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
> >  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
> >  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
> >  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
> >  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
> >  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
> >  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
> >  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
> >  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
> >  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
> >  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
> >  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
> >  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
> >  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
> >  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
> >  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
> >  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
> >  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
> >  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
> >  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
> >  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
> >  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
> >  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
> >  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
> >  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 67
> >
> > Looks like neither of these ksp/pc combos are good? I also tried
> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
> GAMG info I was able to grep from the out-of-box params:
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
> min=2.394056e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 5.21674 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
> min=6.783380e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 9.85289 nnz ave. (N=537)
> > [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
> min=5.974776e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 12.4 nnz ave. (N=100)
> > [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
> min=4.842076e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> >
> > And the one with pc_gamg_agg_nsmooths 0
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with
> threshold 0., 3.07398 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> > [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.02211 nnz ave. (N=814)
> > [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> > [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.00434 nnz ave. (N=461)
> > [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> > [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=290)
> > [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> > [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=197)
> > [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> > [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.98425 nnz ave. (N=127)
> > [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> > [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.97561 nnz ave. (N=82)
> > [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> > [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.9697 nnz ave. (N=66)
> > [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> > [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> >
> >
> > Thanks,
> > Justin
> >
> > On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> >
> >
> > Both GAMG and ILU are nice and dandy for this,
> >
> > I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
> converge it way down, say rtol = 1.e-12. See which one is better in the
> early iteration and pick it. It would be nice to check that it solves the
> problem ...
> >
> > The residual drops about 5 orders in the first iteration and then
> flatlines. That is very bad. Check that the smoother can actually solve the
> problem.
> >
> > but as soon as I look at a bigger system, like a network with 8500
> buses, the out-of-box gamg craps out. I am not sure where to start when it
> comes to tuning GAMG.
> >
> > First try -pc_gamg_nsmooths 0
> >
> > You can run with -info and grep on GAMG to see some diagnostic output.
> Eigen estimates are fragile in practice and with this parameter and SOR
> there are no eigen estimates needed. The max eigen values for all levels
> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
> problem.
> >
> > Attached is the ksp monitor/view output for gamg on the unsuccessful
> solve
> >
> > I'm also attaching a zip file which contains the simple PETSc script
> that loads the binary matrix/vector as well as two test cases, if you guys
> are interested in trying it out. It only works if you have PETSc configured
> with complex numbers.
> >
> > Thanks
> >
> > Justin
> >
> > PS - A couple years ago I had asked if there was a paper/tutorial on
> using/tuning GAMG. Does such a thing exist today?
> >
> > There is a write up in the manual that is tutorial like.
> >
> >
> > On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
> wrote:
> > Here's IMHO the simplest explanation of the equations I'm trying to
> solve:
> >
> > http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> >
> > Right now we're just trying to solve eq(5) (in section 1), inverting the
> linear Y-bus matrix. Eventually we have to be able to solve equations like
> those in the next section.
> >
> > Maybe I am reading this wrong, but the Y-bus matrix looks like an
> M-matrix to me (if all the y's are positive). This means
> > that it should be really easy to solve, and I think GAMG should do it.
> You can start out just doing relaxation, like SOR, on
> > small examples.
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi all,
> >
> > I'm working with some folks to extract a linear system of equations from
> an external software package that solves power flow equations in complex
> form. Since that external package uses serial direct solvers like KLU from
> suitesparse, I want a proof-of-concept where the same matrix can be solved
> in PETSc using its parallel solvers.
> >
> > I got mumps to achieve a very minor speedup across two MPI processes on
> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
> seconds). However I want to use iterative solvers and preconditioners but I
> have never worked with complex numbers so I am not sure what the "best"
> options are given PETSc's capabilities.
> >
> > So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
> believe I also tried BICG with BJACOBI and while it did converge it
> converged slowly. Does anyone have recommendations on how one would go
> about preconditioning PETSc matrices with complex numbers? I was originally
> thinking about converting it to cartesian form: Declaring all voltages =
> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
> arctan(imaginary/real) because all the papers I've seen in literature that
> claim to successfully precondition power flow equations operate in this
> form.
> >
> > 1) We really need to see the (simplified) equations
> >
> > 2) All complex equations can be converted to a system of real equations
> twice as large, but this is not necessarily the best way to go
> >
> >  Thanks,
> >
> >     Matt
> >
> > Justin
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/58028cd6/attachment-0001.html>

From huq2090 at gmail.com  Tue Feb  5 10:36:57 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Tue, 5 Feb 2019 10:36:57 -0600
Subject: [petsc-users] Installing PETSc
In-Reply-To: <87a7ja715b.fsf@jedbrown.org>
References: <CAE5QvP1DpJfMBMRJ3FskfLQF8+WicPTG-jjfDReBZjMj_YbDkA@mail.gmail.com>
	<87a7ja715b.fsf@jedbrown.org>
Message-ID: <CAE5QvP1R7uG3JuObhoSQ+xCOHLFQJZA8gVu0+TO+s3mCHBCe=g@mail.gmail.com>

Thanks!

I got the following values for the variables:

huq2090 at huq2090-XPS-15-9570:~$ echo $PETSC_DIR
/home/huq2090/petsc-3.10.2
huq2090 at huq2090-XPS-15-9570:~$ echo $PETSC_ARCH
arch-linux2-c-debug
huq2090 at huq2090-XPS-15-9570:~$

Does this look correct?

Again, I can run the code now.
For some reason I can't run last day.

Thanks.
Sincerely,
Huq

On Tue, Feb 5, 2019 at 8:36 AM Jed Brown <jed at jedbrown.org> wrote:

> Fazlul Huq via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Hello PETSc Developers,
> >
> > may be this is a trivial question!
> >
> > I usually run PETSc code from Home/petsc-3.10.2 directory. Last day I
> tried
> > to run the code from Documents/petsc directory but I can't. As far as I
> can
> > recall, I have installed PETSc in the Home directory. Is it the reason
> why
> > I can't run PETSc code from other directory? Shall I install PETSc in the
> > root directory?
>
> What do you mean by "run PETSc"?  How did you link your executable?
>
> > Again, if I run command "which petsc" I don't get any echo on the
> terminal.
>
> That is searching for an executable named "petsc" in your PATH.
>
>   echo $PATH
>
> to see the value of that variable.
>


-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/d47c0d1c/attachment.html>

From shrirang.abhyankar at pnnl.gov  Tue Feb  5 11:11:51 2019
From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G)
Date: Tue, 5 Feb 2019 17:11:51 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
Message-ID: <DM6PR09MB324427878046F60A3C60039BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>

Ok, I'll add two things; (i) a TSEventSetPostEventTimeStep(), and (ii) a flag that allows the user to select either the previous time-step or the original time-step. For most users, the flag should suffice. For users who know how the dynamics would behave after the event, they can set the appropriate time-step with TSEventSetPostEventTimeStep().

>>>   It does seem absurd for the code to revert to a tiny step in dealing with the
>>>Event if the time stepper is happy with a much much large time-step for
>>>solving the ODE at that time range.

Once the event is located the system equations get modified that may change the system behavior. As such, the current approach was to use
a conservative smaller time-step. With the above additions, a larger time-step will be allowed.

Shri

>>>-----Original Message-----
>>>From: Smith, Barry F. <bsmith at mcs.anl.gov>
>>>Sent: Monday, February 4, 2019 9:53 PM
>>>To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
>>>Cc: Huck, Moritz <Moritz.Huck at rwth-aachen.de>; PETSc users list <petsc-
>>>users at mcs.anl.gov>
>>>Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>>>
>>>
>>>  Ahh, thanks, for some reason I didn't find that.
>>>
>>>  Based on Moritz's code example perhaps we need to add one or more
>>>TSEvent methods. The simplest thing would be to have a routine to allow
>>>reseting the "base" timestep that is used 1) within the TSEvent search and 2)
>>>after an event is found. Something like
>>>
>>>   TSEventSetBaseTimeStep()
>>>
>>>of course Base is a terrible name but I'm not sure what else to use.
>>>
>>>   This is, of course, not a perfect solution because the user would have to
>>>know what value to use (though they could set it using values from
>>>TSGetTimeStep().)
>>>
>>>   As an alternative or additional feature there could be a flag so that TSEvent
>>>proceeds with the current time step for 1) and 2) not the timestep_orig?
>>>
>>>   It does seem absurd for the code to revert to a tiny step in dealing with the
>>>Event if the time stepper is happy with a much much large time-step for
>>>solving the ODE at that time range.
>>>
>>>  Barry
>>>
>>>
>>>
>>>
>>>
>>>> On Feb 4, 2019, at 7:48 PM, Abhyankar, Shrirang G
>>><shrirang.abhyankar at pnnl.gov> wrote:
>>>>
>>>> It's stashed in event->timestep_orig. At the beginning of the TSEventHandler
>>>there is this code that sets timestep_orig.
>>>>
>>>> if (event->status == TSEVENT_NONE) {
>>>>    if (ts->steps == 1) /* After first successful step */
>>>>      event->timestep_orig = ts->ptime - ts->ptime_prev;
>>>>
>>>> After the step is resynchronized after event location, the next step chosen is
>>>the min. of the time-step before the event interval (event->timestep_prev)
>>>and the original step.
>>>>
>>>> if (event->status == TSEVENT_RESET_NEXTSTEP) {
>>>>    /* Restore time step */
>>>>    dt = PetscMin(event->timestep_orig,event->timestep_prev);
>>>>    ierr = TSSetTimeStep(ts,dt);CHKERRQ(ierr);
>>>>
>>>> So, if the event->timestep_prev is smaller than event->timestep_orig
>>>> (which could be possibly with adaptivity ON, it is allowed)
>>>>
>>>> Shri
>>>>
>>>> -----Original Message-----
>>>> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of
>>>> Smith, Barry F. via petsc-users
>>>> Sent: Monday, February 4, 2019 4:37 PM
>>>> To: Huck, Moritz <Moritz.Huck at rwth-aachen.de>
>>>> Cc: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>;
>>>> petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>>>>
>>>>>
>>>>> Is there any reason you'd like to use the last time step? We used the initial
>>>time-step (which is assumed to be small) so as (a) to minimize any numerical
>>>issues that may be caused by a large last time-step, and (b) to not miss any
>>>ensuing events that may be triggered by the discontinuity.
>>>>>
>>>>> Shri
>>>>
>>>>  Shri
>>>>
>>>>     How do you get access to this initial time-step later in the run?
>>>TSSetTimeStep() sets the value ts->time_step but I don't see any place where
>>>that initial timestep is stashed so could be used later.
>>>>
>>>>   Thanks
>>>>
>>>>    Barry
>>>>
>>>>
>>>>> On Feb 4, 2019, at 1:56 AM, Huck, Moritz via petsc-users <petsc-
>>>users at mcs.anl.gov> wrote:
>>>>>
>>>>> Hi,
>>>>> yes there is one.
>>>>> My system has a set of very different time contants.
>>>>> At some events the fast one may be fully developped and only the slower
>>>ones are of concern after the event (depending on the postevent).
>>>>> If the last step is not an option, setting it manually might be helpfull since I
>>>generall know which timeconstants will be relevant (TSSetTime step seems
>>>not to have an effect during the postevent call).
>>>>>
>>>>> Moritz
>>>>> ________________________________________
>>>>> Von: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
>>>>> Gesendet: Montag, 4. Februar 2019 05:09:24
>>>>> An: Huck, Moritz; Zhang, Hong
>>>>> Betreff: RE: [petsc-users] [TimeStepping] Eventhandler
>>>>>
>>>>> Is there any reason you'd like to use the last time step? We used the initial
>>>time-step (which is assumed to be small) so as (a) to minimize any numerical
>>>issues that may be caused by a large last time-step, and (b) to not miss any
>>>ensuing events that may be triggered by the discontinuity.
>>>>>
>>>>> Shri
>>>>>
>>>>> -----Original Message-----
>>>>> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of
>>>>> Moritz Huck via petsc-users
>>>>> Sent: Sunday, February 3, 2019 10:45 AM
>>>>> To: Zhang, Hong <hongzhang at anl.gov>
>>>>> Cc: petsc-users at mcs.anl.gov
>>>>> Subject: Re: [petsc-users] [TimeStepping] Eventhandler
>>>>>
>>>>> Hi,
>>>>>
>>>>> I see I am sorry, I misinterpreted the output -ts_adapt_monitor.
>>>>>
>>>>> You are right point 2 is not an issue.
>>>>>
>>>>> I there a way to let the time stepping continue with the last time step
>>>instead of the initial time step?
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Moritz
>>>>>
>>>>> Am 03.02.19 um 17:04 schrieb Zhang, Hong:
>>>>>> ex40 seems to work fine. Here is the output with -ts_monitor -
>>>ts_event_monitor -ts_dt 0.01 :
>>>>>>
>>>>>> 8 TS dt 0.5 time 3.11
>>>>>> 9 TS dt 0.5 time 3.61
>>>>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>>>>> 9 TS dt 0.467955 time 3.61 (r)
>>>>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>>>>> [3.61 - 4.07796]
>>>>>> 10 TS dt 0.00365179 time 4.07796
>>>>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>>>>> [4.07796 - 4.08161]
>>>>>> 11 TS dt 2.5403e-05 time 4.08161
>>>>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>>>>>> iterations Ball hit the ground at t =  4.08 seconds
>>>>>> 12 TS dt 0.0283673 time 4.08163
>>>>>> 13 TS dt 0.01 time 4.11
>>>>>> 14 TS dt 0.1 time 4.12
>>>>>>
>>>>>> The event occurs during the step [3.61- 4.11]. After the event is located at
>>>4.08163, the next step after the event should be 4.11-4.08163 = 0.02837 (see
>>>12 TS) to complete the original step. After 4.11, the specified initial stepsize
>>>dt=0.01 will be used.
>>>>>>
>>>>>> Hong (Mr.)
>>>>>>
>>>>>>> On Feb 3, 2019, at 6:08 AM, Moritz Huck via petsc-users <petsc-
>>>users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>> Hi Barry,
>>>>>>>
>>>>>>> src/ts/examples/tutorials/ex40.c (run with ARKIMEX3 and initial dt=1e-
>>>2) has the same behavior:
>>>>>>>
>>>>>>> TSEvent: iter 0 - Event 0 interval detected [3.61 - 4.11]
>>>>>>> TSEvent: iter 1 - Stepping forward as no event detected in interval
>>>>>>> [3.61 - 4.07796]
>>>>>>> TSEvent: iter 2 - Stepping forward as no event detected in interval
>>>>>>> [4.07796 - 4.08161]
>>>>>>> TSEvent: Event 0 zero crossing at time 4.08163 located in 3
>>>>>>> iterations Ball hit the ground at t =  4.08 seconds
>>>>>>>      TSAdapt basic arkimex 0:3 step  13 accepted t=4.11       + 1.000e-02
>>>dt=1.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>>>>>
>>>>>>>      TSAdapt basic arkimex 0:3 step  14 accepted t=4.12       + 1.000e-01
>>>dt=5.000e-01  wlte=    0  wltea=   -1 wlter=   -1
>>>>>>>
>>>>>>> It continues at the end of the "event interval" with the initial timestep.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Moritz
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Am 03.02.19 um 02:43 schrieb Smith, Barry F.:
>>>>>>>>> On Feb 2, 2019, at 12:08 PM, Moritz Huck via petsc-users <petsc-
>>>users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I am trying to use SetEventHandler.
>>>>>>>>> I hav eencountered two "problems":
>>>>>>>>>
>>>>>>>>> 1. after an event is encountered the next time step is needlessly small
>>>(h=1e-6), even if a don't change anything in the function (for testing).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2. the integration does not continue at the event encounter e.g.
>>>>>>>>> my eventfunction is g(t,u)=t-1, the event is found in the step
>>>>>>>>> from t=
>>>>>>>>> 0.9 to 1.1 at tevent=1 but the next step is not 1+dt but it is
>>>>>>>>> 1.1
>>>>>>>>> + dt I would expect the next step to be 1.+dt
>>>>>>>>     Hmm, this sounds like it could be an error. Could you send us a
>>>simple code that exhibits it?
>>>>>>>>
>>>>>>>>   Thanks
>>>>>>>>
>>>>>>>>   Barry
>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Moritz
>>>>>>>>>
>>>>


From Moritz.Huck at rwth-aachen.de  Tue Feb  5 11:30:15 2019
From: Moritz.Huck at rwth-aachen.de (Huck, Moritz)
Date: Tue, 5 Feb 2019 17:30:15 +0000
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <87d0o671f5.fsf@jedbrown.org>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
	<0d77472b87c54dd2a9040612a405888e@rwth-aachen.de>,
	<87d0o671f5.fsf@jedbrown.org>
Message-ID: <e2f018d6ec70473698e6f71079c94aec@rwth-aachen.de>

I am using the fenics library for my PDE discretization.
I can not compile fenics with PETSc 3.10.3.
I will locate the exact error tomorrow.

________________________________________
Von: Jed Brown <jed at jedbrown.org>
Gesendet: Dienstag, 5. Februar 2019 15:30:38
An: Huck, Moritz; Smith, Barry F.; Abhyankar, Shrirang G
Cc: PETSc users list
Betreff: Re: [petsc-users] [TimeStepping] Eventhandler

"Huck, Moritz via petsc-users" <petsc-users at mcs.anl.gov> writes:

> @Shri
> The system is very stiff, but the stiffness is handled well by ARKIMEX.
>
> I'am using PETSc 3.10. (I cannot use 3.10.3 at the moment due to
> compatibilty with a third library),

What compatibility problem is this?  3.10.3 should be (binary and
source) backward compatible with other 3.10 releases, so if there is a
case where it is not, we need to understand it.


Regarding the event, it seems there are scenarios where something
important changes in the model and a conservative time step is
desirable, and other scenarios where the dynamics after the event are on
the same scale as before so it makes sense to continue with the same
step sizes.

From bsmith at mcs.anl.gov  Tue Feb  5 12:18:27 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 5 Feb 2019 18:18:27 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>
	<DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>



> On Feb 5, 2019, at 10:32 AM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> 
> Thanks Barry,
>      I will explore how to partition for parallel computation later. But now I still have some confusion on the sequential operation.
> I compared PETSC and Mumps. In both case, the subroutine for generating elemental matrix is very similar. However, the difference is in the following step?
>    call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> In this step ,each element cost about 3~4 e-2 second.
> 
> However, when I use mumps, I use the following code: 
>    preA(Aptr+1:Aptr+n*(n+1)/2) = pack(Ae, mask(1:n,1:n))
>    Aptr = Aptr +n*(n+1)/2
>    if (allocated(auxRHSe)) preRHS(idx) = preRHS(idx)+auxRHSe
>  It just cost 10e-6~10e-5 second. For a 10000*10000 matrix, the assembling time for using PETSC is 300s while it cost 60s when using Mumps.
> For a 10000*10000 system,the  Is there any way I can make it faster? 

  As we keep saying the slowness of the matrix assembly is due to incorrect preallocation. Once you have the preallocation correct the speed should increase dramatically.

> Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?

   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.

  Barry


> 
> Thanks
> 
> Yaxiong
> 
> 
> 
> 
> 
> Yaxiong Chen, 
> Ph.D. Student 
> 
> School of Mechanical Engineering, 3171 
> 585 Purdue Mall
> West Lafayette, IN 47907
> 
> 
> 
> 
> 
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Monday, February 4, 2019 10:42 PM
> To: Yaxiong Chen
> Cc: PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> 
> 
> > On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > 
> >   Hi Barry,
> > 
> >  !===================================================
> >       mystart =rank +1                                    ! rank starts from 0
> >       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process 
> >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
> >             ne=size(idx)            
> >             idx=idx-1           !-1 since PETSC index starts from zero 
> >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)  
> >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
> >       end do
> > !===================================================
> >   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.
> 
>    This is very bad strategy because there is no data locality. 
> 
> > Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?
> 
>   What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.
> 
>    Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.
> 
>    Barry
> 
> > 
> > Thanks
> > 
> > Yaxiong
> > 
> > 
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Monday, February 4, 2019 5:21 PM
> > To: Yaxiong Chen
> > Cc: Mark Adams; PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > 
> > 
> > > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > Hi Mark,
> > >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> > >       do i=mystart,nelem,nproc
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > >             ne=size(idx)
> > >             idx=idx-1           !-1 since PETSC index starts from zero 
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > >       end do
> > > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> > >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > 
> > > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> > 
> > 
> >    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> > 
> > 
> > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> > >  aseembel time   282.54871800000001     
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > > [0] PCSetUp(): Setting up PC for first time
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > 
> > > However, when I use the parallel mode,I got the following information:
> > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> > 
> >    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> > 
> >   Barry
> > 
> > 
> > 
> > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > 
> > > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > Is there anything I am doing wrong?
> > > Thanks
> > > 
> > > Yaxiong Chen, 
> > > Ph.D. Student 
> > > 
> > > School of Mechanical Engineering, 3171 
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > > 
> > > 
> > > 
> > > 
> > > 
> > > From: Mark Adams <mfadams at lbl.gov>
> > > Sent: Tuesday, January 29, 2019 10:02 PM
> > > To: Yaxiong Chen; PETSc users list
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >  
> > > Optimized is a configuration flag not a versions.
> > > 
> > > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > > 
> > > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > > 
> > > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > Thanks Mark, 
> > >     I use PETSC 3.9.4, is this the optimized version you called?
> > >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > > 
> > > Yaxiong Chen, 
> > > Ph.D. Student 
> > > 
> > > School of Mechanical Engineering, 3171 
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > > 
> > > 
> > > 
> > > 
> > > 
> > > From: Mark Adams <mfadams at lbl.gov>
> > > Sent: Tuesday, January 29, 2019 8:25 PM
> > > To: Yaxiong Chen
> > > Cc: Song Gao; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >  
> > > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > > 
> > > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > > 
> > > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > > 
> > > And I trust you are running an optimized version of PETSc.
> > > 
> > > 
> > > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > Hi Song,
> > > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix? 
> > > <pastedImage.png>
> > > 
> > > 
> > > Thanks
> > > 
> > > Yaxiong Chen, 
> > > Ph.D. Student 
> > > 
> > > School of Mechanical Engineering, 3171 
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > > 
> > > 
> > > 
> > > 
> > > 
> > > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > > Sent: Tuesday, January 29, 2019 1:22 PM
> > > To: Yaxiong Chen
> > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >  
> > > I think you would prefer to preallocate the matrix
> > > 
> > > uncomment this line 
> > > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > 
> > > 
> > > 
> > > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > > Hello,
> > > 
> > > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > > 
> > > Thanks
> > > 
> > > block
> > >     PetscErrorCode ierr
> > >      PetscMPIInt rank,nproc, mystart
> > >      PetscInt  nelem
> > >      integer,allocatable  ::info(:)
> > >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> > >      integer, allocatable      :: idx(:)
> > >      PetscScalar, pointer :: xx_v(:)
> > >   
> > >      PC               prec
> > >      PetscScalar ::   val
> > >      Vec              xvec,bvec,uvec
> > >      Mat              Amat
> > >      KSP              ksp
> > >      PetscViewer viewer
> > >      PetscInt  geq,j,k,ne,M,Istart,Iend
> > >      PetscBool  flg
> > >      KSPConvergedReason reason
> > >      Vec         dummyVec, dummyVecs(1)
> > >      MatNullSpace nullspace
> > > 
> > >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > > 
> > >      if (ierr .ne. 0) then
> > >      print*,'Unable to initialize PETSc'
> > >      stop
> > >      endif
> > >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> > >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > >      mystart=rank+1
> > > 
> > >  !   Parameter set
> > >      info=ptsystem%getInitInfo()
> > >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain) 
> > >      print*,'nelem',nelem
> > >  !-------------------------------------
> > >  !  Create Matrix
> > >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> > >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> > >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> > >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > >      call MatSetFromOptions( Amat, ierr )
> > >      call MatSetUp( Amat, ierr )
> > >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> > >  
> > >       xvec = tVec(0)
> > >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> > >       call VecSetFromOptions( xvec, ierr )
> > >       call VecDuplicate( xvec, bvec, ierr )
> > >       call VecDuplicate( xvec, uvec, ierr )
> > > 
> > >    t1 = MPI_WTime(); 
> > > 
> > >       do i=mystart,nelem,nproc
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > >             ne=size(idx)
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> > >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> > >       end do
> > > 
> > >       nelem   = info(Info_EleRHSNum)
> > >       mystart = rank+1
> > > 
> > >       do i = mystart, nelem, nproc  
> > >         call ptSystem%getElementalRHS(i, RHSe, idx)
> > >         print*,'idx',idx
> > >         ne=size(idx)
> > >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> > >       end do 
> > >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> > >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> > >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> > >      t2 = MPI_WTime(); 
> > >       print*,'assembeling time',t2-t1
> > >       ! Solve
> > >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > > !  Set operators. Here the matrix that defines the linear system
> > > !  also serves as the preconditioning matrix.
> > >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> > >       call KSPSetFromOptions(ksp,ierr)
> > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > !                      Solve the linear system
> > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > >     call KSPSetType(ksp,KSPCG,ierr)
> > >     call KSPGetPC(ksp,prec,ierr)
> > >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> > >     call PCSetType(prec,PCJACOBI,ierr)
> > >      call KSPSolve(ksp,bvec,xvec,ierr)
> > >      call PetscFinalize(ierr)
> > > 
> > > end block
> > > 
> > > 
> > > Yaxiong Chen, 
> > > Ph.D. Student 
> > > 
> > > School of Mechanical Engineering, 3171 
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907


From knepley at gmail.com  Tue Feb  5 13:01:29 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 Feb 2019 14:01:29 -0500
Subject: [petsc-users] MWE for DMPlexCreateCGNS
In-Reply-To: <CAHJ1_RMd=kcYAV7DuXwgC8svKLHf_0cOYUGj7BP-8FjAYDpFOA@mail.gmail.com>
References: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>
	<CAMYG4GkocLUpbcfrKGywqxqHJUq4thARMV4c5BzV-9Xv2apfhQ@mail.gmail.com>
	<CAHJ1_RMd=kcYAV7DuXwgC8svKLHf_0cOYUGj7BP-8FjAYDpFOA@mail.gmail.com>
Message-ID: <CAMYG4GmeRRB=A5MtRcU_KWtuB+t7Or2UJzoapDp2366Af_Lwww@mail.gmail.com>

On Tue, Feb 5, 2019 at 11:13 AM Andrew Parker <andy.john.parker at gmail.com>
wrote:

> On Tue, 5 Feb 2019 at 15:27, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Feb 5, 2019 at 9:47 AM Andrew Parker via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Does anyone have a MWE for DMPlexCreateCGNS to use in parallel? Ideally,
>>> read parallel, distribute in parallel, construct ghost cells (for parallel
>>> comms and halos for physical boundaries)? It's for a cell-centered solver
>>> working with cgns meshes.  Is there any limitation on cell-types?
>>>
>>
>> 1) CGNS is a terrible format, unfortunately. Are you sure about
>> committing to it? Always best to question design decisions before lots of
>> code has been written.
>>
>
>> 2) DMPlexCreateCGNS() probably works, but it has not been tested
>> exhaustively.
>>
>
>> 3) It does not work in parallel, and is unlikely to in the near future.
>> In order to read in parallel, you should be able to nicely select
>>     blocks from disk in the format, and also have local BC specification
>> (rather than global numbers). MED is a nice format like this.
>>     It is the CASCADE format, and GMsh uses it internally. We can now
>> read MED in parallel. Without too much work, we could also
>>     read ExodusII in parallel I think.
>>
>
> Do you have a MWE for either of those two formats then?
>
> Key for me is to be in parallel by the time the setup stage is finished,
> if there is some serial reading here and there at the initial phase that's
> fine.  For a cell-centered code I just want to go from file read to
> parallel distribute, parallel ghost-cell creation, physical halo creation,
> and boundary condition (faces, edge in 2D) marking re-using as much of the
> petsc DMPlex system as possible.
>

Oh, that is fine then.


> My main requirement with any file format is the ability in a mesh/cad
> package to mark complex groups of edges/faces/cells together (on the
> boundary but also groupings of cells in the volume) to infer groups and
> therefore bcs, or material regions etc.  I am aware med/exo provide this:
> my reason for picking cgns is that is central to their format also.  I need
> to ensure for complex geometry and meshes, that if the file format does
> encode groupings/bcs then these are accessible after the parallel
> distribution phase in complete.  It would help me massively if there were
> an example that does all of this....I'll jump ship on file format to get
> that.
>

I think it should be fine to use CGNS for you. How about you make a small
CGNS file with some marked boundaries
and I will help get that in as a PETSc example (we can use Plex ex1 to read
it in and write out the Plex).

  Thanks,

     Matt


> Thanks,
> Andy
>
>>
>> 4) Once read, you get a regular Plex, so you can redistribute, make ghost
>> cells, etc. as you can for any Plex mesh.
>>
>> 5) As long as all you want Plex to do is manage topology and field data,
>> then you can have whatever mix of cell types you want.
>>
>> 6) Limitations on cell types come from routines that calculate geometric
>> quantities, integrate, etc.
>>
>>   Thanks,
>>
>>      Matt
>>
>
>
>>
>>
>>> Thanks,
>>> Andy
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/9feb6727/attachment-0001.html>

From andy.john.parker at gmail.com  Tue Feb  5 13:07:06 2019
From: andy.john.parker at gmail.com (Andrew Parker)
Date: Tue, 5 Feb 2019 19:07:06 +0000
Subject: [petsc-users] MWE for DMPlexCreateCGNS
In-Reply-To: <CAMYG4GmeRRB=A5MtRcU_KWtuB+t7Or2UJzoapDp2366Af_Lwww@mail.gmail.com>
References: <CAHJ1_RONaYPr6oKVmJeM=NNOKd0ruMwp2oZA4LVQ6DSqVU=M8A@mail.gmail.com>
	<CAMYG4GkocLUpbcfrKGywqxqHJUq4thARMV4c5BzV-9Xv2apfhQ@mail.gmail.com>
	<CAHJ1_RMd=kcYAV7DuXwgC8svKLHf_0cOYUGj7BP-8FjAYDpFOA@mail.gmail.com>
	<CAMYG4GmeRRB=A5MtRcU_KWtuB+t7Or2UJzoapDp2366Af_Lwww@mail.gmail.com>
Message-ID: <CAHJ1_RNqTyFkBQbYdm+SBpakxGireQOxdC0=N_75axXwsxAnLA@mail.gmail.com>

Will do, I'll get back to you.
Cheers,
Andy

On Tue, 5 Feb 2019 at 19:01, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Feb 5, 2019 at 11:13 AM Andrew Parker <andy.john.parker at gmail.com>
> wrote:
>
>> On Tue, 5 Feb 2019 at 15:27, Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Tue, Feb 5, 2019 at 9:47 AM Andrew Parker via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Does anyone have a MWE for DMPlexCreateCGNS to use in parallel?
>>>> Ideally, read parallel, distribute in parallel, construct ghost cells (for
>>>> parallel comms and halos for physical boundaries)? It's for a cell-centered
>>>> solver working with cgns meshes.  Is there any limitation on cell-types?
>>>>
>>>
>>> 1) CGNS is a terrible format, unfortunately. Are you sure about
>>> committing to it? Always best to question design decisions before lots of
>>> code has been written.
>>>
>>
>>> 2) DMPlexCreateCGNS() probably works, but it has not been tested
>>> exhaustively.
>>>
>>
>>> 3) It does not work in parallel, and is unlikely to in the near future.
>>> In order to read in parallel, you should be able to nicely select
>>>     blocks from disk in the format, and also have local BC specification
>>> (rather than global numbers). MED is a nice format like this.
>>>     It is the CASCADE format, and GMsh uses it internally. We can now
>>> read MED in parallel. Without too much work, we could also
>>>     read ExodusII in parallel I think.
>>>
>>
>> Do you have a MWE for either of those two formats then?
>>
>> Key for me is to be in parallel by the time the setup stage is finished,
>> if there is some serial reading here and there at the initial phase that's
>> fine.  For a cell-centered code I just want to go from file read to
>> parallel distribute, parallel ghost-cell creation, physical halo creation,
>> and boundary condition (faces, edge in 2D) marking re-using as much of the
>> petsc DMPlex system as possible.
>>
>
> Oh, that is fine then.
>
>
>> My main requirement with any file format is the ability in a mesh/cad
>> package to mark complex groups of edges/faces/cells together (on the
>> boundary but also groupings of cells in the volume) to infer groups and
>> therefore bcs, or material regions etc.  I am aware med/exo provide this:
>> my reason for picking cgns is that is central to their format also.  I need
>> to ensure for complex geometry and meshes, that if the file format does
>> encode groupings/bcs then these are accessible after the parallel
>> distribution phase in complete.  It would help me massively if there were
>> an example that does all of this....I'll jump ship on file format to get
>> that.
>>
>
> I think it should be fine to use CGNS for you. How about you make a small
> CGNS file with some marked boundaries
> and I will help get that in as a PETSc example (we can use Plex ex1 to
> read it in and write out the Plex).
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Andy
>>
>>>
>>> 4) Once read, you get a regular Plex, so you can redistribute, make
>>> ghost cells, etc. as you can for any Plex mesh.
>>>
>>> 5) As long as all you want Plex to do is manage topology and field data,
>>> then you can have whatever mix of cell types you want.
>>>
>>> 6) Limitations on cell types come from routines that calculate geometric
>>> quantities, integrate, etc.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>
>>
>>>
>>>
>>>> Thanks,
>>>> Andy
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/30e89e64/attachment.html>

From mfadams at lbl.gov  Tue Feb  5 14:22:03 2019
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 5 Feb 2019 15:22:03 -0500
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <CADOhEh7swaLGqofUWAa9SLF7MpV7va8tWFmgvBHDuvRsCMGnzQ@mail.gmail.com>

On Mon, Feb 4, 2019 at 4:17 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:

> Hi Mark,
>
>      Will the parameter MatMPIAIJSetPreallocation in influence the
> following part
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             idx=idx-1           !-1 since PETSC index starts from zero
>             if (allocated(auxRHSe))  call
> VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
>             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
>       end do
>
> I found this part will impede my assembling process. In my case ,the total
> DOF is 20800. I estimated the upper bound  of number of nonzero entries in
> each row as 594.
>

Then you want to set f9 to 594


> So I set f9 to be 20206 and f6 to be 10103 in the following command:
>
 call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>
>
> Running the code in sequential mode with -info I got the following
> information.And I can get the result even thought the assembling process is
> kind of slow(several times slower than using Mumps).
>
> [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
>
> [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
>
> [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1;
> storage space: 13847 unneeded, 1030038 used
>
> [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is
> 62659
>
> [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
>
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
>
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472
> 140509341618528
>
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472
> 140509341618528
>
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472
> 140509341618528
>
> [0] VecScatterCreate_Seq(): Special case: sequential copy
>
> [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1;
> storage space: 104000 unneeded, 0 used
>

This looks like your matrix was create with 0 columns.


> [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
>
> [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
>
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
>
>  aseembel time   282.54871800000001
>
> [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960
> 140509341442256
>
> [0] PCSetUp(): Setting up PC for first time
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> [0] PCSetUp(): Leaving PC with identical preconditioner since operator is
> unchanged
>
> However, when I use the parallel mode,I got the following information:
>
> [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
>
> [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
>
> it seems it never went to  call
> MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
>
> Is there anything I am doing wrong?
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Tuesday, January 29, 2019 10:02 PM
> *To:* Yaxiong Chen; PETSc users list
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> Optimized is a configuration flag not a versions.
>
> You need to figure out your number of non-zeros per row of you global
> matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation.
> Otherwise it has to allocate and copy memory often.
>
> You could increase your f9 on a serial run and see what runs best and
> then  move to parallel with a value in f6 of about 1/2 of f9.
>
> On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Thanks Mark,
>
>     I use PETSC 3.9.4, is this the optimized version you called?
>
>     Actually f9 and f6 are from the PETSC example. I don't know how to set
> the value correctly so I commend them. The size of my elemental matrix may
> vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or
> some other sizes. And the index is not continuous. In this case, the
> elemental matrix may interlace with each other in the global matrix, and I
> may have thousands of elemental matrix to be assembled. Does the
> preallocating suitable for this?
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Tuesday, January 29, 2019 8:25 PM
> *To:* Yaxiong Chen
> *Cc:* Song Gao; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> Slow assembly is often from not preallocating correctly. I am guessing
> that you are using Q1 element and f9==9, and thus the preallocation should
> be OK if this is a scalar problem on a regular grid and f6-==6 should be OK
> for the off processor allocation, if my assumptions are correct.
>
> You can run with -info, which will tell you how many allocation were done
> in assembly. Make sure that it is small (eg, 0).
>
> I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy
> things with seeming simple code. You could just do this manually if you can
> find anything else.
>
> And I trust you are running an optimized version of PETSc.
>
>
> On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi Song,
>
> I don't quite understand how I can use this command. I don't partition the
> gloabl matrix. If I add my elemental matrix to the global system it will be
> like this. And in my parallel part, I use each core to generate the
> elemental matrix in turn. In this case, I guess each core will be assigned
> the space for global matrix and finally be assembled.But according to the
> manual, it seems each core will store a part of the global matrix. Is the local
> submatrix in the MatMPIAIJSetPreallocation
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>
> (Mat
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/Mat.html#Mat>
> B,PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> d_nnz[],PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nz,const PetscInt
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>
> o_nnz[])the same as my elemental matrix?
>
>
>
> Thanks
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
>
> ------------------------------
> *From:* Song Gao <song.gao2 at mail.mcgill.ca>
> *Sent:* Tuesday, January 29, 2019 1:22 PM
> *To:* Yaxiong Chen
> *Cc:* Matthew Knepley; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] PETSC matrix assembling super slow
>
> I think you would prefer to preallocate the matrix
>
> uncomment this line
> ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>
>
>
> Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <
> petsc-users at mcs.anl.gov> a ?crit :
>
> Hello,
>
>
> I have a 2D system which is assembled by each elemental matrix.  Ae is my
> elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side,
> idx is the global index. My code is as follow, however ,the assembling rate
> is super slow(Marked red in the code). I am not sure whether the assembling
> type is right or not. Since for each element, idx are not continuous
> numbers. Do you have any idea what is the better way to assemble the matrix?
>
>
> Thanks
>
>
> block
>     PetscErrorCode ierr
>      PetscMPIInt rank,nproc, mystart
>      PetscInt  nelem
>      integer,allocatable  ::info(:)
>      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
>      integer, allocatable      :: idx(:)
>      PetscScalar, pointer :: xx_v(:)
>
>      PC               prec
>      PetscScalar ::   val
>      Vec              xvec,bvec,uvec
>      Mat              Amat
>      KSP              ksp
>      PetscViewer viewer
>      PetscInt  geq,j,k,ne,M,Istart,Iend
>      PetscBool  flg
>      KSPConvergedReason reason
>      Vec         dummyVec, dummyVecs(1)
>      MatNullSpace nullspace
>
>      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
>
>      if (ierr .ne. 0) then
>      print*,'Unable to initialize PETSc'
>      stop
>      endif
>      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
>      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
>      mystart=rank+1
>
>  !   Parameter set
>      info=ptsystem%getInitInfo()
>      nelem =
> info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
>      print*,'nelem',nelem
>  !-------------------------------------
>  !  Create Matrix
>      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
>      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1),
> ierr )
>      call MatSetType( Amat, MATMPIBAIJ, ierr )
>      ! call
> MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER,
> ierr)
>      call MatSetFromOptions( Amat, ierr )
>      call MatSetUp( Amat, ierr )
>      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
>
>       xvec = tVec(0)
>       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
>       call VecSetFromOptions( xvec, ierr )
>       call VecDuplicate( xvec, bvec, ierr )
>       call VecDuplicate( xvec, uvec, ierr )
>
>    t1 = MPI_WTime();
>
>       do i=mystart,nelem,nproc
>             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
>             ne=size(idx)
>             if (allocated(auxRHSe))  call
> VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
>              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
>       end do
>
>       nelem   = info(Info_EleRHSNum)
>       mystart = rank+1
>
>       do i = mystart, nelem, nproc
>         call ptSystem%getElementalRHS(i, RHSe, idx)
>         print*,'idx',idx
>         ne=size(idx)
>         if (allocated(RHSe))  call
> VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
>       end do
>       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
>       !  this part is slow, the for loop above is done but here it may get
> stuck
>       call VecAssemblyBegin(bvec,ierr)
>                                 ! For a 2500 DOF system, assembling only
> takes over 2 seconds
>       call VecAssemblyEnd(bvec,ierr)
>                                 ! But for a 10000 DOF system , it gets stuck
>      t2 = MPI_WTime();
>       print*,'assembeling time',t2-t1
>       ! Solve
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>       call KSPSetOperators(ksp,Amat,Amat,ierr)
>       call KSPSetFromOptions(ksp,ierr)
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>     call KSPSetType(ksp,KSPCG,ierr)
>     call KSPGetPC(ksp,prec,ierr)
>    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
>     call PCSetType(prec,PCJACOBI,ierr)
>      call KSPSolve(ksp,bvec,xvec,ierr)
>      call PetscFinalize(ierr)
>
> end block
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
>
> 585 Purdue Mall
>
> West Lafayette, IN 47907
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/52de4ff3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 3840 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/52de4ff3/attachment-0001.png>

From Sanjay.Kharche at lhsc.on.ca  Tue Feb  5 17:06:30 2019
From: Sanjay.Kharche at lhsc.on.ca (Sanjay Kharche)
Date: Tue, 5 Feb 2019 23:06:30 +0000
Subject: [petsc-users] reading petsc binary files.
Message-ID: <0dff9cee1c044728974dcf4462f39597@lhsc.on.ca>


Hi

I use two mpi clusters (cluster 1 and 2). Whereas the petsc binary files I generate can be read on cluster 1, I get errors doing so on cluster 2. I also output vts files corresponding to each binary file output, and it appears that both clusters do produce meaningful results. I use ver 3.7.4 on cluster 1, and 3.7.5 on 2. Also, the same simulation produces binary files of slightly different sizes on the two clusters. Can you comment on what I need to do to be able to read binary files on cluster 2.

thanks
Sanjay

Code snippet from parallel petsc code that does output:

if(time_int % (int)(1.0/DELTAT) == 0){ // a smaller time step and more files outputted makes the CV estimate better.
sprintf(str,"my_2d%d.vts", file_Counter); // this confirms that simulation does something meaningful.
PetscViewer viewer;  PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERVTK);
PetscViewerFileSetName(viewer, str); VecView(u, viewer); PetscViewerDestroy(&viewer);
                 sprintf(str,"my_3d%d.bin",(int)file_Counter);
PetscViewer viewer2;
PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE,&viewer2);
VecView(u,viewer2);
PetscViewerDestroy(&viewer2);
file_Counter++;
}


How I am trying to read it (typically serial code with binary called ecg):

   // inputs are Petsc binary files, create and destroy viewer at each file for simplicity.
PetscViewer viewer_in;
sprintf(str,"my_3d%d.bin",file_Counter);
PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_READ,&viewer_in);
VecLoad(u,viewer_in);
PetscViewerDestroy(&viewer_in);


Errors I got when I ran ecg:

login3 endo]$ ./ecg
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Not a vector next in file
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017
[0]PETSC ERROR: ./ecg on a arch-linux2-c-opt named gra-login3 by kharches Tue Feb  5 17:43:36 2019
[0]PETSC ERROR: Configure options --prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/petsc/3.7.5 --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18 --download-hypre=1 --download-metis=1 --download-triangle=1 --download-ptscotch=1 --download-superlu_dist=1 --download-ml=1 --download-superlu=1 --download-prometheus=1 --download-mumps=1 --download-parmetis=1 --download-suitesparse=1 --download-mumps-shared=0 --download-ptscotch-shared=0 --download-superlu-shared=0 --download-superlu_dist-shared=0 --download-parmetis-shared=0 --download-metis-shared=0 --download-ml-shared=0 --download-suitesparse-shared=0 --download-hypre-shared=0 --download-prometheus-shared=0 --with-cc=mpicc --with-cxx=mpicxx --with-c++-support --with-fc=mpifort --CFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --CXXFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --FFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --with-gnu-compilers=0 --with-mpi=1 --with-build-step-np=8 --with-shared-libraries=1 --with-debugging=0 --with-pic=1 --with-x=0 --with-windows-graphics=0 --with-scalapack=1 --with-scalapack-include=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/include --with-scalapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_scalapack_lp64.a,libmkl_blacs_openmpi_lp64.a,libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]" --with-blas-lapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]" --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18 --with-fftw=1 --with-fftw-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/fftw-mpi/3.3.6
[0]PETSC ERROR: #1 PetscViewerBinaryReadVecHeader_Private() line 28 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #2 VecLoad_Binary() line 90 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #3 VecLoad_Default() line 413 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
[0]PETSC ERROR: #4 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #5 VecLoad_Binary_DA() line 931 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
[0]PETSC ERROR: #6 VecLoad_Default_DA() line 964 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
[0]PETSC ERROR: #7 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.

From jed at jedbrown.org  Tue Feb  5 17:11:16 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 Feb 2019 16:11:16 -0700
Subject: [petsc-users] [TimeStepping] Eventhandler
In-Reply-To: <e2f018d6ec70473698e6f71079c94aec@rwth-aachen.de>
References: <e843438f-18be-ee53-9b42-f573d211d924@rwth-aachen.de>
	<D9DCE4E7-C6E9-46DA-907D-4B6C9BE8AA25@anl.gov>
	<b0d49dd6-d139-7ec7-da47-fe554cd99989@rwth-aachen.de>
	<A4940052-A3C8-4142-ABE7-A837477EC251@anl.gov>
	<2bf10c74-46c7-e834-0b71-e0931d467e41@rwth-aachen.de>
	<DM6PR09MB32448EF7037AA806C766B14DE76D0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<660c205547424a4db43951a5bc1fce38@rwth-aachen.de>
	<A75CBEA1-853A-4E96-9662-B3C3FC275323@anl.gov>
	<DM6PR09MB32447459BE6D5024F599E25BE76E0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<9878EECD-37B1-4932-9C53-680EA52D28C0@mcs.anl.gov>
	<0d77472b87c54dd2a9040612a405888e@rwth-aachen.de>
	<87d0o671f5.fsf@jedbrown.org>
	<e2f018d6ec70473698e6f71079c94aec@rwth-aachen.de>
Message-ID: <871s4lu8yz.fsf@jedbrown.org>

Dolfin will need this PR to work with any PETSc 3.10.

https://bitbucket.org/fenics-project/dolfin/pull-requests/508/jed-petsc-310/diff

It's been chillin' there for a couple months; it appears that much of
the Dolfin development effort has moved to DolfinX and Firedrake.

"Huck, Moritz" <Moritz.Huck at rwth-aachen.de> writes:

> I am using the fenics library for my PDE discretization.
> I can not compile fenics with PETSc 3.10.3.
> I will locate the exact error tomorrow.
>
> ________________________________________
> Von: Jed Brown <jed at jedbrown.org>
> Gesendet: Dienstag, 5. Februar 2019 15:30:38
> An: Huck, Moritz; Smith, Barry F.; Abhyankar, Shrirang G
> Cc: PETSc users list
> Betreff: Re: [petsc-users] [TimeStepping] Eventhandler
>
> "Huck, Moritz via petsc-users" <petsc-users at mcs.anl.gov> writes:
>
>> @Shri
>> The system is very stiff, but the stiffness is handled well by ARKIMEX.
>>
>> I'am using PETSc 3.10. (I cannot use 3.10.3 at the moment due to
>> compatibilty with a third library),
>
> What compatibility problem is this?  3.10.3 should be (binary and
> source) backward compatible with other 3.10 releases, so if there is a
> case where it is not, we need to understand it.
>
>
> Regarding the event, it seems there are scenarios where something
> important changes in the model and a conservative time step is
> desirable, and other scenarios where the dynamics after the event are on
> the same scale as before so it makes sense to continue with the same
> step sizes.

From jychang48 at gmail.com  Tue Feb  5 17:22:52 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 5 Feb 2019 16:22:52 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
Message-ID: <CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>

Hi all,

So I *think* I know what the problem is. I am modeling a a distribution
system where each bus can have anywhere from 1 to 3 phases. In the smallest
example I had, all four buses had 3 phases (hence 12 dofs), and this was
very easy to decompose - each MPI process should have a specific set of
complex voltages on their respective nodes/buses. The larger test cases
have lots of single phase buses attached, so a naive decomposition of the
matrix isn't trivial. Again we're only extracting the Y-bus matrices out of
an external software (OpenDSS to be specific) so we have no a priori
information on how they organize their degrees-of-freedom - we let
MatLoad() and VecLoad() do the decomposition. So it's possible that during
this decomposition, a single bus' phases are being split across two or more
MPI processes, which I assume would mess up the algorithmic performance of
GAMG/ASM when more than one MPI process is needed.

Yeah DMNetwork will fix this and ensure that all dofs per vertex do not get
split across different MPI procs, but for now we're just trying to see if
we can get a simple proof-of-concept working. We're trying to come up with
a toy distribution system where everything is three-phase (hence very
straightforward to decompose even with MatLoad().

Justin

On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov> wrote:

> I would stay away from eigen estimates in the solver (but give us the
> spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.
>
> Applications that have lived on direct solvers can add sorts of crap like
> penalty terms.
>
> sor seemed to work OK so I'd check the coarse grids in GAMG. Test with
> just two levels. That way you don't have to use sor on an internal MG grid,
> which can be bad and probably is here.
>
>
> On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>
>>
>> > On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Thanks everyone for your suggestions/feedback. So a few things:
>> >
>> > 1) When I examined larger distribution networks (~150k buses) my
>> eigenvalue estimates from the chebyshev method get enormous indeed. See
>> below:
>> >
>> > [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
>> nnz/row (ave)=6, np=1
>> > [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with
>> threshold 0., 6.01293 nnz ave. (N=320745)
>> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> > [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
>> min=4.639523e-02 PC=jacobi
>> > [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1
>> active pes
>> > [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with
>> threshold 0., 7.32435 nnz ave. (N=44797)
>> > [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
>> min=9.687506e-01 PC=jacobi
>> > [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1
>> active pes
>> > [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with
>> threshold 0., 22.5607 nnz ave. (N=13043)
>> > [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
>> min=6.150856e-01 PC=jacobi
>> > [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1
>> active pes
>> > [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with
>> threshold 0., 79.5142 nnz ave. (N=2256)
>> > [0] PCGAMGProlongator_AGG(): New grid 232 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
>> min=6.780909e-01 PC=jacobi
>> > [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1
>> active pes
>> > [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with
>> threshold 0., 206.379 nnz ave. (N=232)
>> > [0] PCGAMGProlongator_AGG(): New grid 9 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
>> min=2.153627e-01 PC=jacobi
>> > [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
>> > [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
>> >
>> > 2) I tried all the suggestions mentioned before: setting
>> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
>> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
>> iterations of richardson/sor.
>> >
>> > 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm
>> just solving LU on the whole system.
>> >
>> > 3b)The problem is that I can't get it to even speedup across 2 MPI
>> processes for my 150k bus case (~300k dofs) - and I already checked that
>> this problem could in theory be parallelized by setting the -ksp_max_it to
>> something low and observing that the KSPSolve time decreases as MPI
>> concurrency increases. The potential speedup is countered by the fact that
>> the algorithmic convergence rate blows up when either more MPI processes
>> are added or when I tune the block_size/overlap parameters.
>>
>>    Yes, we just haven't gotten a scalable preconditioner yet for your
>> matrix.
>>
>> >
>> > Which leads me to one of two conclusions/questions:
>> >
>> > A) Is my 300 DOF problem still too small? Do I need to look at a
>> problem with, say, 10 Million DOF or more to see that increasing the
>> asm_block_size will have better performance (despite having more KSP
>> iterations) than a single giant asm_block?
>>
>>    No, if you had a scalable preconditioner you would see good speed up
>> with 300k unknowns, using 10 million won't help.
>>
>> >
>> > B) Or is there something even more sinister about my system of
>> equations in complex form?
>>
>>   There is a technical term for matrices for which both GAMG and ASM do
>> very poorly on: nasty :-)
>>
>>   Have you tried using parallel LU (for example ./configure
>> --download-superlu_dist). Direct LU is the last refuge of the
>> unpreconditionable. How large will your matrices get?
>>
>>    Barry
>>
>> >
>> > Thanks,
>> > Justin
>> >
>> > On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > The coarse grid is getting set to SOR rather than LU.
>> >
>> >    Matt
>> >
>> > On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com>
>> wrote:
>> > I tried these options:
>> >
>> > -ksp_view
>> > -ksp_monitor_true_residual
>> > -ksp_type gmres
>> > -ksp_max_it 50
>> > -pc_type gamg
>> > -mg_coarse_pc_type sor
>> > -mg_levels_1_ksp_type richardson
>> > -mg_levels_2_ksp_type richardson
>> > -mg_levels_3_ksp_type richardson
>> > -mg_levels_4_ksp_type richardson
>> > -mg_levels_1_pc_type sor
>> > -mg_levels_2_pc_type sor
>> > -mg_levels_3_pc_type sor
>> > -mg_levels_4_pc_type sor
>> >
>> > And still have a non-converging solution:
>> >
>> >   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
>> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>> >   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
>> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>> >   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
>> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>> >   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
>> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>> >   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
>> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>> >   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
>> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>> >   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
>> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>> >   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
>> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>> >   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
>> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>> >   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
>> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>> >  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
>> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>> >  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
>> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>> >  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
>> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>> >  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
>> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>> >  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
>> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>> >  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
>> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>> >  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
>> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>> >  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
>> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>> >  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
>> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>> >  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
>> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>> >  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
>> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>> >  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
>> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>> >  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
>> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>> >  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
>> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>> >  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
>> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>> >  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
>> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>> >  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
>> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>> >  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
>> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>> >  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
>> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>> >  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
>> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>> >  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
>> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>> >  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
>> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>> >  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
>> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>> >  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
>> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>> >  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
>> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>> >  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
>> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>> >  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
>> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>> >  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
>> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>> >  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
>> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>> >  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
>> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>> >  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
>> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>> >  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
>> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>> >  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
>> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>> >  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
>> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>> >  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
>> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>> >  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
>> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>> >  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
>> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>> >  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
>> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>> >  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
>> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>> >  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
>> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>> >  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
>> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
>> > KSP Object: 1 MPI processes
>> >   type: gmres
>> >     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >     happy breakdown tolerance 1e-30
>> >   maximum iterations=50, initial guess is zero
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >   left preconditioning
>> >   using PRECONDITIONED norm type for convergence test
>> > PC Object: 1 MPI processes
>> >   type: gamg
>> >     type is MULTIPLICATIVE, levels=5 cycles=v
>> >       Cycles per PCApply=1
>> >       Using externally compute Galerkin coarse grid matrices
>> >       GAMG specific options
>> >         Threshold for dropping small values in graph on each level =
>>  0.   0.   0.
>> >         Threshold scaling factor for each level not specified = 1.
>> >         AGG specific options
>> >           Symmetric graph false
>> >           Number of levels to square graph 1
>> >           Number smoothing steps 1
>> >         Complexity:    grid = 1.31821
>> >   Coarse grid solver -- level -------------------------------
>> >     KSP Object: (mg_coarse_) 1 MPI processes
>> >       type: preonly
>> >       maximum iterations=10000, initial guess is zero
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >     PC Object: (mg_coarse_) 1 MPI processes
>> >       type: sor
>> >         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>> >       linear system matrix = precond matrix:
>> >       Mat Object: 1 MPI processes
>> >         type: seqaij
>> >         rows=17, cols=17
>> >         total: nonzeros=163, allocated nonzeros=163
>> >         total number of mallocs used during MatSetValues calls =0
>> >           using I-node routines: found 12 nodes, limit used is 5
>> >   Down solver (pre-smoother) on level 1 -------------------------------
>> >     KSP Object: (mg_levels_1_) 1 MPI processes
>> >       type: richardson
>> >         damping factor=1.
>> >       maximum iterations=2, nonzero initial guess
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >     PC Object: (mg_levels_1_) 1 MPI processes
>> >       type: sor
>> >         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>> >       linear system matrix = precond matrix:
>> >       Mat Object: 1 MPI processes
>> >         type: seqaij
>> >         rows=100, cols=100
>> >         total: nonzeros=1240, allocated nonzeros=1240
>> >         total number of mallocs used during MatSetValues calls =0
>> >           not using I-node routines
>> >   Up solver (post-smoother) same as down solver (pre-smoother)
>> >   Down solver (pre-smoother) on level 2 -------------------------------
>> >     KSP Object: (mg_levels_2_) 1 MPI processes
>> >       type: richardson
>> >         damping factor=1.
>> >       maximum iterations=2, nonzero initial guess
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >     PC Object: (mg_levels_2_) 1 MPI processes
>> >       type: sor
>> >         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>> >       linear system matrix = precond matrix:
>> >       Mat Object: 1 MPI processes
>> >         type: seqaij
>> >         rows=537, cols=537
>> >         total: nonzeros=5291, allocated nonzeros=5291
>> >         total number of mallocs used during MatSetValues calls =0
>> >           not using I-node routines
>> >   Up solver (post-smoother) same as down solver (pre-smoother)
>> >   Down solver (pre-smoother) on level 3 -------------------------------
>> >     KSP Object: (mg_levels_3_) 1 MPI processes
>> >       type: richardson
>> >         damping factor=1.
>> >       maximum iterations=2, nonzero initial guess
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >     PC Object: (mg_levels_3_) 1 MPI processes
>> >       type: sor
>> >         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>> >       linear system matrix = precond matrix:
>> >       Mat Object: 1 MPI processes
>> >         type: seqaij
>> >         rows=1541, cols=1541
>> >         total: nonzeros=8039, allocated nonzeros=8039
>> >         total number of mallocs used during MatSetValues calls =0
>> >           not using I-node routines
>> >   Up solver (post-smoother) same as down solver (pre-smoother)
>> >   Down solver (pre-smoother) on level 4 -------------------------------
>> >     KSP Object: (mg_levels_4_) 1 MPI processes
>> >       type: richardson
>> >         damping factor=1.
>> >       maximum iterations=2, nonzero initial guess
>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>> >       left preconditioning
>> >       using NONE norm type for convergence test
>> >     PC Object: (mg_levels_4_) 1 MPI processes
>> >       type: sor
>> >         type = local_symmetric, iterations = 1, local iterations = 1,
>> omega = 1.
>> >       linear system matrix = precond matrix:
>> >       Mat Object: 1 MPI processes
>> >         type: seqaij
>> >         rows=8541, cols=8541
>> >         total: nonzeros=46299, allocated nonzeros=46299
>> >         total number of mallocs used during MatSetValues calls =0
>> >           using I-node routines: found 5464 nodes, limit used is 5
>> >   Up solver (post-smoother) same as down solver (pre-smoother)
>> >   linear system matrix = precond matrix:
>> >   Mat Object: 1 MPI processes
>> >     type: seqaij
>> >     rows=8541, cols=8541
>> >     total: nonzeros=46299, allocated nonzeros=46299
>> >     total number of mallocs used during MatSetValues calls =0
>> >       using I-node routines: found 5464 nodes, limit used is 5
>> >
>> > Am I doing this right? Did I miss anything?
>> >
>> >
>> >
>> > On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com>
>> wrote:
>> > Hi Mark,
>> >
>> > 1) So with these options:
>> >
>> > -ksp_type gmres
>> > -ksp_rtol 1e-15
>> > -ksp_monitor_true_residual
>> > -ksp_converged_reason
>> > -pc_type bjacobi
>> >
>> > This is what I get:
>> >
>> >   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>> >   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
>> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>> >   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
>> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>> >   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
>> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>> >   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
>> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>> >   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
>> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
>> > Linear solve converged due to CONVERGED_RTOL iterations 5
>> >
>> > 2) With richardson/sor:
>> >
>> > Okay, its looks like Richardson/SOR solves this just fine. You can use
>> this as the smoother for GAMG instead
>> > of Cheby/Jacobi, and probably see better results on the larger problems.
>> >
>> >   Matt
>> >
>> > -ksp_type richardson
>> > -ksp_rtol 1e-15
>> > -ksp_monitor_true_residual
>> > -pc_type sor
>> >
>> > This is what I get:
>> >
>> >   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
>> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>> >   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
>> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>> >   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
>> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>> >   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
>> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>> >   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
>> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>> >   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
>> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>> >   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
>> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>> >   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
>> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>> >   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
>> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>> >   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
>> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>> >  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
>> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>> >  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
>> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>> >  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
>> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>> >  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
>> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>> >  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
>> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>> >  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
>> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>> >  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
>> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>> >  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
>> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>> >  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
>> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>> >  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
>> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>> >  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
>> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>> >  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
>> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>> >  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
>> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>> >  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
>> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>> >  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
>> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>> >  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
>> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>> >  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
>> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>> >  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
>> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>> >  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
>> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>> >  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
>> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>> >  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
>> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>> >  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
>> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>> >  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
>> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>> >  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
>> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>> >  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
>> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>> >  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
>> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>> >  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
>> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>> >  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
>> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>> >  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
>> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>> >  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
>> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>> >  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
>> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
>> > Linear solve converged due to CONVERGED_RTOL iterations 40
>> >
>> > 3) And lastly with chebyshev/jacobi:
>> >
>> > -ksp_type chebyshev
>> > -ksp_rtol 1e-15
>> > -ksp_monitor_true_residual
>> > -pc_type jacobi
>> >
>> >   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
>> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>> >   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
>> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>> >   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
>> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>> >   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
>> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>> >   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
>> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>> >   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
>> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>> >   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
>> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>> >   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
>> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>> >   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
>> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>> >   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
>> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>> >  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
>> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>> >  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
>> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>> >  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
>> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>> >  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
>> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>> >  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
>> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>> >  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
>> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>> >  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
>> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>> >  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
>> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>> >  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
>> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>> >  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
>> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>> >  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
>> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>> >  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
>> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>> >  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
>> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>> >  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
>> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>> >  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
>> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>> >  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
>> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>> >  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
>> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>> >  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
>> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>> >  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
>> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>> >  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
>> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>> >  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
>> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>> >  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
>> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>> >  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
>> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>> >  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
>> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>> >  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
>> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>> >  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
>> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>> >  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
>> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>> >  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
>> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>> >  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
>> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>> >  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
>> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>> >  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
>> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>> >  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
>> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>> >  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
>> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>> >  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
>> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>> >  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
>> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>> >  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
>> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>> >  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
>> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>> >  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
>> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>> >  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
>> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>> >  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
>> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>> >  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
>> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>> >  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
>> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>> >  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
>> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>> >  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
>> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>> >  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
>> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>> >  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
>> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>> >  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
>> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>> >  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
>> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>> >  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
>> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>> >  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
>> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>> >  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
>> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>> >  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
>> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>> >  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
>> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>> >  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
>> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>> >  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
>> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>> >  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
>> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>> >  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
>> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
>> > Linear solve converged due to CONVERGED_RTOL iterations 67
>> >
>> > Looks like neither of these ksp/pc combos are good? I also tried
>> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
>> GAMG info I was able to grep from the out-of-box params:
>> >
>> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> nnz/row (ave)=5, np=1
>> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
>> threshold 0., 5.42079 nnz ave. (N=8541)
>> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
>> min=2.394056e-02 PC=jacobi
>> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 5.21674 nnz ave. (N=1541)
>> > [0] PCGAMGProlongator_AGG(): New grid 537 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
>> min=6.783380e-02 PC=jacobi
>> > [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 9.85289 nnz ave. (N=537)
>> > [0] PCGAMGProlongator_AGG(): New grid 100 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
>> min=5.974776e-02 PC=jacobi
>> > [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 12.4 nnz ave. (N=100)
>> > [0] PCGAMGProlongator_AGG(): New grid 17 nodes
>> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
>> min=4.842076e-01 PC=jacobi
>> > [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active
>> pes
>> > [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>> >
>> > And the one with pc_gamg_agg_nsmooths 0
>> >
>> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
>> nnz/row (ave)=5, np=1
>> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
>> threshold 0., 5.42079 nnz ave. (N=8541)
>> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
>> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with
>> threshold 0., 3.07398 nnz ave. (N=1541)
>> > [0] PCGAMGProlongator_AGG(): New grid 814 nodes
>> > [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 3.02211 nnz ave. (N=814)
>> > [0] PCGAMGProlongator_AGG(): New grid 461 nodes
>> > [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 3.00434 nnz ave. (N=461)
>> > [0] PCGAMGProlongator_AGG(): New grid 290 nodes
>> > [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 3. nnz ave. (N=290)
>> > [0] PCGAMGProlongator_AGG(): New grid 197 nodes
>> > [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 3. nnz ave. (N=197)
>> > [0] PCGAMGProlongator_AGG(): New grid 127 nodes
>> > [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 2.98425 nnz ave. (N=127)
>> > [0] PCGAMGProlongator_AGG(): New grid 82 nodes
>> > [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 2.97561 nnz ave. (N=82)
>> > [0] PCGAMGProlongator_AGG(): New grid 66 nodes
>> > [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active
>> pes
>> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with
>> threshold 0., 2.9697 nnz ave. (N=66)
>> > [0] PCGAMGProlongator_AGG(): New grid 36 nodes
>> > [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active
>> pes
>> > [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>> >
>> >
>> > Thanks,
>> > Justin
>> >
>> > On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
>> >
>> >
>> > Both GAMG and ILU are nice and dandy for this,
>> >
>> > I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
>> converge it way down, say rtol = 1.e-12. See which one is better in the
>> early iteration and pick it. It would be nice to check that it solves the
>> problem ...
>> >
>> > The residual drops about 5 orders in the first iteration and then
>> flatlines. That is very bad. Check that the smoother can actually solve the
>> problem.
>> >
>> > but as soon as I look at a bigger system, like a network with 8500
>> buses, the out-of-box gamg craps out. I am not sure where to start when it
>> comes to tuning GAMG.
>> >
>> > First try -pc_gamg_nsmooths 0
>> >
>> > You can run with -info and grep on GAMG to see some diagnostic output.
>> Eigen estimates are fragile in practice and with this parameter and SOR
>> there are no eigen estimates needed. The max eigen values for all levels
>> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
>> problem.
>> >
>> > Attached is the ksp monitor/view output for gamg on the unsuccessful
>> solve
>> >
>> > I'm also attaching a zip file which contains the simple PETSc script
>> that loads the binary matrix/vector as well as two test cases, if you guys
>> are interested in trying it out. It only works if you have PETSc configured
>> with complex numbers.
>> >
>> > Thanks
>> >
>> > Justin
>> >
>> > PS - A couple years ago I had asked if there was a paper/tutorial on
>> using/tuning GAMG. Does such a thing exist today?
>> >
>> > There is a write up in the manual that is tutorial like.
>> >
>> >
>> > On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
>> wrote:
>> > Here's IMHO the simplest explanation of the equations I'm trying to
>> solve:
>> >
>> > http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>> >
>> > Right now we're just trying to solve eq(5) (in section 1), inverting
>> the linear Y-bus matrix. Eventually we have to be able to solve equations
>> like those in the next section.
>> >
>> > Maybe I am reading this wrong, but the Y-bus matrix looks like an
>> M-matrix to me (if all the y's are positive). This means
>> > that it should be really easy to solve, and I think GAMG should do it.
>> You can start out just doing relaxation, like SOR, on
>> > small examples.
>> >
>> >   Thanks,
>> >
>> >     Matt
>> >
>> > On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > Hi all,
>> >
>> > I'm working with some folks to extract a linear system of equations
>> from an external software package that solves power flow equations in
>> complex form. Since that external package uses serial direct solvers like
>> KLU from suitesparse, I want a proof-of-concept where the same matrix can
>> be solved in PETSc using its parallel solvers.
>> >
>> > I got mumps to achieve a very minor speedup across two MPI processes on
>> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
>> seconds). However I want to use iterative solvers and preconditioners but I
>> have never worked with complex numbers so I am not sure what the "best"
>> options are given PETSc's capabilities.
>> >
>> > So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
>> believe I also tried BICG with BJACOBI and while it did converge it
>> converged slowly. Does anyone have recommendations on how one would go
>> about preconditioning PETSc matrices with complex numbers? I was originally
>> thinking about converting it to cartesian form: Declaring all voltages =
>> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
>> arctan(imaginary/real) because all the papers I've seen in literature that
>> claim to successfully precondition power flow equations operate in this
>> form.
>> >
>> > 1) We really need to see the (simplified) equations
>> >
>> > 2) All complex equations can be converted to a system of real equations
>> twice as large, but this is not necessarily the best way to go
>> >
>> >  Thanks,
>> >
>> >     Matt
>> >
>> > Justin
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190205/5e971fbe/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Feb  5 17:36:24 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 5 Feb 2019 23:36:24 +0000
Subject: [petsc-users] reading petsc binary files.
In-Reply-To: <0dff9cee1c044728974dcf4462f39597@lhsc.on.ca>
References: <0dff9cee1c044728974dcf4462f39597@lhsc.on.ca>
Message-ID: <64428CEE-5452-42C8-B4D1-84FD2E2EAEA2@anl.gov>



> On Feb 5, 2019, at 5:06 PM, Sanjay Kharche via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> Hi
> 
> I use two mpi clusters (cluster 1 and 2). Whereas the petsc binary files I generate can be read on cluster 1, I get errors doing so on cluster 2. I also output vts files corresponding to each binary file output, and it appears that both clusters do produce meaningful results. I use ver 3.7.4 on cluster 1, and 3.7.5 on 2. Also, the same simulation produces binary files of slightly different sizes on the two clusters.

   This is truly strange. PETSc has always used the exact same format for its binary files and automatically handles any byte-swapping that may be need to always have a consistent representation on the disk. Also a difference in the file size is strange. In over 20 years we've never had an issue with unreadable binary files.

  Can cluster 1 read the binary files from cluster 1 and cluster 2 read the binary files from cluster 2. Is it just cluster 2 cannot read the files from cluster 1? 

  Could the file be changed somehow as it is copied between the clusters?

  Is this difference in size reproducible if you delete the files and create them again?

  Is one of the clusters by any chance a Microsoft Windows cluster?

   Barry

> Can you comment on what I need to do to be able to read binary files on cluster 2.
> 
> thanks
> Sanjay
> 
> Code snippet from parallel petsc code that does output:
> 
> if(time_int % (int)(1.0/DELTAT) == 0){ // a smaller time step and more files outputted makes the CV estimate better.
> sprintf(str,"my_2d%d.vts", file_Counter); // this confirms that simulation does something meaningful.
> PetscViewer viewer;  PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERVTK);
> PetscViewerFileSetName(viewer, str); VecView(u, viewer); PetscViewerDestroy(&viewer);
>                 sprintf(str,"my_3d%d.bin",(int)file_Counter);
> PetscViewer viewer2;
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE,&viewer2);
> VecView(u,viewer2);
> PetscViewerDestroy(&viewer2);
> file_Counter++;
> }
> 
> 
> How I am trying to read it (typically serial code with binary called ecg):
> 
>   // inputs are Petsc binary files, create and destroy viewer at each file for simplicity.
> PetscViewer viewer_in;
> sprintf(str,"my_3d%d.bin",file_Counter);
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_READ,&viewer_in);
> VecLoad(u,viewer_in);
> PetscViewerDestroy(&viewer_in);
> 
> 
> Errors I got when I ran ecg:
> 
> login3 endo]$ ./ecg
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Not a vector next in file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017
> [0]PETSC ERROR: ./ecg on a arch-linux2-c-opt named gra-login3 by kharches Tue Feb  5 17:43:36 2019
> [0]PETSC ERROR: Configure options --prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/petsc/3.7.5 --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18 --download-hypre=1 --download-metis=1 --download-triangle=1 --download-ptscotch=1 --download-superlu_dist=1 --download-ml=1 --download-superlu=1 --download-prometheus=1 --download-mumps=1 --download-parmetis=1 --download-suitesparse=1 --download-mumps-shared=0 --download-ptscotch-shared=0 --download-superlu-shared=0 --download-superlu_dist-shared=0 --download-parmetis-shared=0 --download-metis-shared=0 --download-ml-shared=0 --download-suitesparse-shared=0 --download-hypre-shared=0 --download-prometheus-shared=0 --with-cc=mpicc --with-cxx=mpicxx --with-c++-support --with-fc=mpifort --CFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --CXXFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --FFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --with-gnu-compilers=0 --with-mpi=1 --with-build-step-np=8 --with-shared-libraries=1 --with-debugging=0 --with-pic=1 --with-x=0 --with-windows-graphics=0 --with-scalapack=1 --with-scalapack-include=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/include --with-scalapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_scalapack_lp64.a,libmkl_blacs_openmpi_lp64.a,libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]" --with-blas-lapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]" --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18 --with-fftw=1 --with-fftw-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/fftw-mpi/3.3.6
> [0]PETSC ERROR: #1 PetscViewerBinaryReadVecHeader_Private() line 28 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #2 VecLoad_Binary() line 90 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #3 VecLoad_Default() line 413 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #4 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #5 VecLoad_Binary_DA() line 931 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> [0]PETSC ERROR: #6 VecLoad_Default_DA() line 964 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> [0]PETSC ERROR: #7 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
> 
> This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.


From shrirang.abhyankar at pnnl.gov  Tue Feb  5 19:09:13 2019
From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G)
Date: Wed, 6 Feb 2019 01:09:13 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
Message-ID: <DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>

Hi Justin,
   Typically, the power grid distribution systems have a radial structure (unless it is an urban area) that leads to a, more or less, staircase type matrix. So a MatLoad() or VecLoad() would presumably just just splits the stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the phases for a bus to be split across processors, but I don?t think it would cause a big performance hit.
I?ve worked with OpenDSS recently and have created a C wrapper for it that you can use to query values from it. I can help out with your experimentations, if you need.

Shri
From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Justin Chang via petsc-users
Sent: Tuesday, February 5, 2019 5:23 PM
To: Mark Adams <mfadams at lbl.gov>
Cc: petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

Hi all,

So I *think* I know what the problem is. I am modeling a a distribution system where each bus can have anywhere from 1 to 3 phases. In the smallest example I had, all four buses had 3 phases (hence 12 dofs), and this was very easy to decompose - each MPI process should have a specific set of complex voltages on their respective nodes/buses. The larger test cases have lots of single phase buses attached, so a naive decomposition of the matrix isn't trivial. Again we're only extracting the Y-bus matrices out of an external software (OpenDSS to be specific) so we have no a priori information on how they organize their degrees-of-freedom - we let MatLoad() and VecLoad() do the decomposition. So it's possible that during this decomposition, a single bus' phases are being split across two or more MPI processes, which I assume would mess up the algorithmic performance of GAMG/ASM when more than one MPI process is needed.

Yeah DMNetwork will fix this and ensure that all dofs per vertex do not get split across different MPI procs, but for now we're just trying to see if we can get a simple proof-of-concept working. We're trying to come up with a toy distribution system where everything is three-phase (hence very straightforward to decompose even with MatLoad().

Justin

On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
I would stay away from eigen estimates in the solver (but give us the spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.

Applications that have lived on direct solvers can add sorts of crap like penalty terms.

sor seemed to work OK so I'd check the coarse grids in GAMG. Test with just two levels. That way you don't have to use sor on an internal MG grid, which can be bad and probably is here.


On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


> On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Thanks everyone for your suggestions/feedback. So a few things:
>
> 1) When I examined larger distribution networks (~150k buses) my eigenvalue estimates from the chebyshev method get enormous indeed. See below:
>
> [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1, nnz/row (ave)=6, np=1
> [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with threshold 0., 6.01293 nnz ave. (N=320745)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00 min=4.639523e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
> [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with threshold 0., 7.32435 nnz ave. (N=44797)
> [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00 min=9.687506e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active pes
> [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with threshold 0., 22.5607 nnz ave. (N=13043)
> [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00 min=6.150856e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
> [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with threshold 0., 79.5142 nnz ave. (N=2256)
> [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00 min=6.780909e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
> [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with threshold 0., 206.379 nnz ave. (N=232)
> [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00 min=2.153627e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
>
> 2) I tried all the suggestions mentioned before: setting -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more iterations of richardson/sor.
>
> 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just solving LU on the whole system.
>
> 3b)The problem is that I can't get it to even speedup across 2 MPI processes for my 150k bus case (~300k dofs) - and I already checked that this problem could in theory be parallelized by setting the -ksp_max_it to something low and observing that the KSPSolve time decreases as MPI concurrency increases. The potential speedup is countered by the fact that the algorithmic convergence rate blows up when either more MPI processes are added or when I tune the block_size/overlap parameters.

   Yes, we just haven't gotten a scalable preconditioner yet for your matrix.

>
> Which leads me to one of two conclusions/questions:
>
> A) Is my 300 DOF problem still too small? Do I need to look at a problem with, say, 10 Million DOF or more to see that increasing the asm_block_size will have better performance (despite having more KSP iterations) than a single giant asm_block?

   No, if you had a scalable preconditioner you would see good speed up with 300k unknowns, using 10 million won't help.

>
> B) Or is there something even more sinister about my system of equations in complex form?

  There is a technical term for matrices for which both GAMG and ASM do very poorly on: nasty :-)

  Have you tried using parallel LU (for example ./configure --download-superlu_dist). Direct LU is the last refuge of the unpreconditionable. How large will your matrices get?

   Barry

>
> Thanks,
> Justin
>
> On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> The coarse grid is getting set to SOR rather than LU.
>
>    Matt
>
> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> I tried these options:
>
> -ksp_view
> -ksp_monitor_true_residual
> -ksp_type gmres
> -ksp_max_it 50
> -pc_type gamg
> -mg_coarse_pc_type sor
> -mg_levels_1_ksp_type richardson
> -mg_levels_2_ksp_type richardson
> -mg_levels_3_ksp_type richardson
> -mg_levels_4_ksp_type richardson
> -mg_levels_1_pc_type sor
> -mg_levels_2_pc_type sor
> -mg_levels_3_pc_type sor
> -mg_levels_4_pc_type sor
>
> And still have a non-converging solution:
>
>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=5 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.   0.   0.
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>         Complexity:    grid = 1.31821
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=17, cols=17
>         total: nonzeros=163, allocated nonzeros=163
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 12 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=100, cols=100
>         total: nonzeros=1240, allocated nonzeros=1240
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=537, cols=537
>         total: nonzeros=5291, allocated nonzeros=5291
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=1541, cols=1541
>         total: nonzeros=8039, allocated nonzeros=8039
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=8541, cols=8541
>         total: nonzeros=46299, allocated nonzeros=46299
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 5464 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=8541, cols=8541
>     total: nonzeros=46299, allocated nonzeros=46299
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 5464 nodes, limit used is 5
>
> Am I doing this right? Did I miss anything?
>
>
>
> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Hi Mark,
>
> 1) So with these options:
>
> -ksp_type gmres
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -ksp_converged_reason
> -pc_type bjacobi
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> Linear solve converged due to CONVERGED_RTOL iterations 5
>
> 2) With richardson/sor:
>
> Okay, its looks like Richardson/SOR solves this just fine. You can use this as the smoother for GAMG instead
> of Cheby/Jacobi, and probably see better results on the larger problems.
>
>   Matt
>
> -ksp_type richardson
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type sor
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> Linear solve converged due to CONVERGED_RTOL iterations 40
>
> 3) And lastly with chebyshev/jacobi:
>
> -ksp_type chebyshev
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type jacobi
>
>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> Linear solve converged due to CONVERGED_RTOL iterations 67
>
> Looks like neither of these ksp/pc combos are good? I also tried -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the GAMG info I was able to grep from the out-of-box params:
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00 min=2.394056e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 5.21674 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00 min=6.783380e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 9.85289 nnz ave. (N=537)
> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00 min=5.974776e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 12.4 nnz ave. (N=100)
> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00 min=4.842076e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>
> And the one with pc_gamg_agg_nsmooths 0
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with threshold 0., 3.07398 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.02211 nnz ave. (N=814)
> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.00434 nnz ave. (N=461)
> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=290)
> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=197)
> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.98425 nnz ave. (N=127)
> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.97561 nnz ave. (N=82)
> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.9697 nnz ave. (N=66)
> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>
>
> Thanks,
> Justin
>
> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
>
>
> Both GAMG and ILU are nice and dandy for this,
>
> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and converge it way down, say rtol = 1.e-12. See which one is better in the early iteration and pick it. It would be nice to check that it solves the problem ...
>
> The residual drops about 5 orders in the first iteration and then flatlines. That is very bad. Check that the smoother can actually solve the problem.
>
> but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG.
>
> First try -pc_gamg_nsmooths 0
>
> You can run with -info and grep on GAMG to see some diagnostic output. Eigen estimates are fragile in practice and with this parameter and SOR there are no eigen estimates needed. The max eigen values for all levels should be between say 2 (or less) and say 3-4. Much higher is a sign of a problem.
>
> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
>
> I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
>
> Thanks
>
> Justin
>
> PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
>
> There is a write up in the manual that is tutorial like.
>
>
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>
> Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
>
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> small examples.
>
>   Thanks,
>
>     Matt
>
> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> Hi all,
>
> I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers.
>
> I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
>
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
>
> 1) We really need to see the (simplified) equations
>
> 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
>
>  Thanks,
>
>     Matt
>
> Justin
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/d2cd3fab/attachment-0001.html>

From chen2018 at purdue.edu  Tue Feb  5 21:29:09 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Wed, 6 Feb 2019 03:29:09 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>
	<DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>
Message-ID: <DM5PR22MB026540243541443935ACA7A1A86F0@DM5PR22MB0265.namprd22.prod.outlook.com>

> Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?

<<   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly <<the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your <<problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.


 << Barry


It seems I can only work with Jacobi pre-conditioner. When I try LU or Cholesky ,I got the error :

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers

[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.9.4, Sep, 11, 2018

[0]PETSC ERROR: ./optimal_mechanical_part.hdc on a arch-darwin-c-debug named hidac.ecn.purdue.edu by chen2018 Tue Feb  5 22:19:08 2019

[0]PETSC ERROR: Configure options

[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/matrix.c

[0]PETSC ERROR: #2 PCSetUp_Cholesky() line 86 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/impls/factor/cholesky/cholesky.c

[0]PETSC ERROR: #3 PCSetUp() line 923 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: #4 KSPSetUp() line 381 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #5 KSPSolve() line 612 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c

 solve time   3.8589999999985025E-003

 reason           0

 It asks me to download the package. Do I want to run the command in the folder of PETSC? But I am not sure what the name of the package should be . I tried with ./configure --download-<PCLU> But it does not work.

________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Tuesday, February 5, 2019 1:18 PM
To: Yaxiong Chen
Cc: PETSc users list
Subject: Re: [petsc-users] PETSC matrix assembling super slow



> On Feb 5, 2019, at 10:32 AM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> Thanks Barry,
>      I will explore how to partition for parallel computation later. But now I still have some confusion on the sequential operation.
> I compared PETSC and Mumps. In both case, the subroutine for generating elemental matrix is very similar. However, the difference is in the following step?
>    call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> In this step ,each element cost about 3~4 e-2 second.
>
> However, when I use mumps, I use the following code:
>    preA(Aptr+1:Aptr+n*(n+1)/2) = pack(Ae, mask(1:n,1:n))
>    Aptr = Aptr +n*(n+1)/2
>    if (allocated(auxRHSe)) preRHS(idx) = preRHS(idx)+auxRHSe
>  It just cost 10e-6~10e-5 second. For a 10000*10000 matrix, the assembling time for using PETSC is 300s while it cost 60s when using Mumps.
> For a 10000*10000 system,the  Is there any way I can make it faster?

  As we keep saying the slowness of the matrix assembly is due to incorrect preallocation. Once you have the preallocation correct the speed should increase dramatically.

> Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?

   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.

  Barry


>
> Thanks
>
> Yaxiong
>
>
>
>
>
> Yaxiong Chen,
> Ph.D. Student
>
> School of Mechanical Engineering, 3171
> 585 Purdue Mall
> West Lafayette, IN 47907
>
>
>
>
>
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Monday, February 4, 2019 10:42 PM
> To: Yaxiong Chen
> Cc: PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>
>
>
> > On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> >
> >   Hi Barry,
> >
> >  !===================================================
> >       mystart =rank +1                                    ! rank starts from 0
> >       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process
> >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
> >             ne=size(idx)
> >             idx=idx-1           !-1 since PETSC index starts from zero
> >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
> >       end do
> > !===================================================
> >   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.
>
>    This is very bad strategy because there is no data locality.
>
> > Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?
>
>   What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.
>
>    Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.
>
>    Barry
>
> >
> > Thanks
> >
> > Yaxiong
> >
> >
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Monday, February 4, 2019 5:21 PM
> > To: Yaxiong Chen
> > Cc: Mark Adams; PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >
> >
> >
> > > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > >
> > > Hi Mark,
> > >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> > >       do i=mystart,nelem,nproc
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > >             ne=size(idx)
> > >             idx=idx-1           !-1 since PETSC index starts from zero
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > >       end do
> > > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> > >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > >
> > > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> >
> >
> >    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> >
> >
> > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> > >  aseembel time   282.54871800000001
> > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > > [0] PCSetUp(): Setting up PC for first time
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > >
> > > However, when I use the parallel mode,I got the following information:
> > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> >
> >    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> >
> >   Barry
> >
> >
> >
> > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > >
> > > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > Is there anything I am doing wrong?
> > > Thanks
> > >
> > > Yaxiong Chen,
> > > Ph.D. Student
> > >
> > > School of Mechanical Engineering, 3171
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > >
> > >
> > >
> > >
> > >
> > > From: Mark Adams <mfadams at lbl.gov>
> > > Sent: Tuesday, January 29, 2019 10:02 PM
> > > To: Yaxiong Chen; PETSc users list
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >
> > > Optimized is a configuration flag not a versions.
> > >
> > > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > >
> > > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > >
> > > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > Thanks Mark,
> > >     I use PETSC 3.9.4, is this the optimized version you called?
> > >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > >
> > > Yaxiong Chen,
> > > Ph.D. Student
> > >
> > > School of Mechanical Engineering, 3171
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > >
> > >
> > >
> > >
> > >
> > > From: Mark Adams <mfadams at lbl.gov>
> > > Sent: Tuesday, January 29, 2019 8:25 PM
> > > To: Yaxiong Chen
> > > Cc: Song Gao; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >
> > > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > >
> > > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > >
> > > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > >
> > > And I trust you are running an optimized version of PETSc.
> > >
> > >
> > > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > Hi Song,
> > > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix?
> > > <pastedImage.png>
> > >
> > >
> > > Thanks
> > >
> > > Yaxiong Chen,
> > > Ph.D. Student
> > >
> > > School of Mechanical Engineering, 3171
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > >
> > >
> > >
> > >
> > >
> > > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > > Sent: Tuesday, January 29, 2019 1:22 PM
> > > To: Yaxiong Chen
> > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >
> > > I think you would prefer to preallocate the matrix
> > >
> > > uncomment this line
> > > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > >
> > >
> > >
> > > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > > Hello,
> > >
> > > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > >
> > > Thanks
> > >
> > > block
> > >     PetscErrorCode ierr
> > >      PetscMPIInt rank,nproc, mystart
> > >      PetscInt  nelem
> > >      integer,allocatable  ::info(:)
> > >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> > >      integer, allocatable      :: idx(:)
> > >      PetscScalar, pointer :: xx_v(:)
> > >
> > >      PC               prec
> > >      PetscScalar ::   val
> > >      Vec              xvec,bvec,uvec
> > >      Mat              Amat
> > >      KSP              ksp
> > >      PetscViewer viewer
> > >      PetscInt  geq,j,k,ne,M,Istart,Iend
> > >      PetscBool  flg
> > >      KSPConvergedReason reason
> > >      Vec         dummyVec, dummyVecs(1)
> > >      MatNullSpace nullspace
> > >
> > >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > >
> > >      if (ierr .ne. 0) then
> > >      print*,'Unable to initialize PETSc'
> > >      stop
> > >      endif
> > >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> > >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > >      mystart=rank+1
> > >
> > >  !   Parameter set
> > >      info=ptsystem%getInitInfo()
> > >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
> > >      print*,'nelem',nelem
> > >  !-------------------------------------
> > >  !  Create Matrix
> > >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> > >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> > >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> > >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > >      call MatSetFromOptions( Amat, ierr )
> > >      call MatSetUp( Amat, ierr )
> > >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> > >
> > >       xvec = tVec(0)
> > >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> > >       call VecSetFromOptions( xvec, ierr )
> > >       call VecDuplicate( xvec, bvec, ierr )
> > >       call VecDuplicate( xvec, uvec, ierr )
> > >
> > >    t1 = MPI_WTime();
> > >
> > >       do i=mystart,nelem,nproc
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > >             ne=size(idx)
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> > >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> > >       end do
> > >
> > >       nelem   = info(Info_EleRHSNum)
> > >       mystart = rank+1
> > >
> > >       do i = mystart, nelem, nproc
> > >         call ptSystem%getElementalRHS(i, RHSe, idx)
> > >         print*,'idx',idx
> > >         ne=size(idx)
> > >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> > >       end do
> > >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> > >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> > >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> > >      t2 = MPI_WTime();
> > >       print*,'assembeling time',t2-t1
> > >       ! Solve
> > >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > > !  Set operators. Here the matrix that defines the linear system
> > > !  also serves as the preconditioning matrix.
> > >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> > >       call KSPSetFromOptions(ksp,ierr)
> > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > !                      Solve the linear system
> > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > >     call KSPSetType(ksp,KSPCG,ierr)
> > >     call KSPGetPC(ksp,prec,ierr)
> > >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> > >     call PCSetType(prec,PCJACOBI,ierr)
> > >      call KSPSolve(ksp,bvec,xvec,ierr)
> > >      call PetscFinalize(ierr)
> > >
> > > end block
> > >
> > >
> > > Yaxiong Chen,
> > > Ph.D. Student
> > >
> > > School of Mechanical Engineering, 3171
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/1a59f332/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Feb  5 21:58:26 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 6 Feb 2019 03:58:26 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB026540243541443935ACA7A1A86F0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>
	<DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>
	<DM5PR22MB026540243541443935ACA7A1A86F0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <8A05158C-E6BF-4663-9188-60BE3CD7FEB2@mcs.anl.gov>


  Run ./configure with --download-superlu_dist --download-mumps --download-scalapack and the you can use either -pc_factor_mat_solver_type superlu_dist or -pc_factor_mat_solver_type mumps

  Good luck


> On Feb 5, 2019, at 9:29 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> 
> > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
> 
> <<   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly <<the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your <<problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
> 
>  << Barry
> 
> 
> It seems I can only work with Jacobi pre-conditioner. When I try LU or Cholesky ,I got the error :
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.4, Sep, 11, 2018 
> [0]PETSC ERROR: ./optimal_mechanical_part.hdc on a arch-darwin-c-debug named hidac.ecn.purdue.edu by chen2018 Tue Feb  5 22:19:08 2019
> [0]PETSC ERROR: Configure options 
> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_Cholesky() line 86 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/impls/factor/cholesky/cholesky.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 381 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 612 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
>  solve time   3.8589999999985025E-003
>  reason           0
>  It asks me to download the package. Do I want to run the command in the folder of PETSC? But I am not sure what the name of the package should be . I tried with ./configure --download-<PCLU> But it does not work.
> 
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, February 5, 2019 1:18 PM
> To: Yaxiong Chen
> Cc: PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> 
> 
> > On Feb 5, 2019, at 10:32 AM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > 
> > Thanks Barry,
> >      I will explore how to partition for parallel computation later. But now I still have some confusion on the sequential operation.
> > I compared PETSC and Mumps. In both case, the subroutine for generating elemental matrix is very similar. However, the difference is in the following step?
> >    call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > In this step ,each element cost about 3~4 e-2 second.
> > 
> > However, when I use mumps, I use the following code: 
> >    preA(Aptr+1:Aptr+n*(n+1)/2) = pack(Ae, mask(1:n,1:n))
> >    Aptr = Aptr +n*(n+1)/2
> >    if (allocated(auxRHSe)) preRHS(idx) = preRHS(idx)+auxRHSe
> >  It just cost 10e-6~10e-5 second. For a 10000*10000 matrix, the assembling time for using PETSC is 300s while it cost 60s when using Mumps.
> > For a 10000*10000 system,the  Is there any way I can make it faster? 
> 
>   As we keep saying the slowness of the matrix assembly is due to incorrect preallocation. Once you have the preallocation correct the speed should increase dramatically.
> 
> > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
> 
>    It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
> 
>   Barry
> 
> 
> > 
> > Thanks
> > 
> > Yaxiong
> > 
> > 
> > 
> > 
> > 
> > Yaxiong Chen, 
> > Ph.D. Student 
> > 
> > School of Mechanical Engineering, 3171 
> > 585 Purdue Mall
> > West Lafayette, IN 47907
> > 
> > 
> > 
> > 
> > 
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Monday, February 4, 2019 10:42 PM
> > To: Yaxiong Chen
> > Cc: PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > 
> > 
> > > On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > 
> > >   Hi Barry,
> > > 
> > >  !===================================================
> > >       mystart =rank +1                                    ! rank starts from 0
> > >       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process 
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
> > >             ne=size(idx)            
> > >             idx=idx-1           !-1 since PETSC index starts from zero 
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)  
> > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
> > >       end do
> > > !===================================================
> > >   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.
> > 
> >    This is very bad strategy because there is no data locality. 
> > 
> > > Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?
> > 
> >   What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.
> > 
> >    Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.
> > 
> >    Barry
> > 
> > > 
> > > Thanks
> > > 
> > > Yaxiong
> > > 
> > > 
> > > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > > Sent: Monday, February 4, 2019 5:21 PM
> > > To: Yaxiong Chen
> > > Cc: Mark Adams; PETSc users list
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >  
> > > 
> > > 
> > > > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > > 
> > > > Hi Mark,
> > > >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> > > >       do i=mystart,nelem,nproc
> > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > >             ne=size(idx)
> > > >             idx=idx-1           !-1 since PETSC index starts from zero 
> > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > > >       end do
> > > > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> > > >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > > 
> > > > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> > > 
> > > 
> > >    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> > > 
> > > 
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> > > >  aseembel time   282.54871800000001     
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > > > [0] PCSetUp(): Setting up PC for first time
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > 
> > > > However, when I use the parallel mode,I got the following information:
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> > > 
> > >    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> > > 
> > >   Barry
> > > 
> > > 
> > > 
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > > 
> > > > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > > Is there anything I am doing wrong?
> > > > Thanks
> > > > 
> > > > Yaxiong Chen, 
> > > > Ph.D. Student 
> > > > 
> > > > School of Mechanical Engineering, 3171 
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > From: Mark Adams <mfadams at lbl.gov>
> > > > Sent: Tuesday, January 29, 2019 10:02 PM
> > > > To: Yaxiong Chen; PETSc users list
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >  
> > > > Optimized is a configuration flag not a versions.
> > > > 
> > > > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > > > 
> > > > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > > > 
> > > > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > > Thanks Mark, 
> > > >     I use PETSC 3.9.4, is this the optimized version you called?
> > > >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > > > 
> > > > Yaxiong Chen, 
> > > > Ph.D. Student 
> > > > 
> > > > School of Mechanical Engineering, 3171 
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > From: Mark Adams <mfadams at lbl.gov>
> > > > Sent: Tuesday, January 29, 2019 8:25 PM
> > > > To: Yaxiong Chen
> > > > Cc: Song Gao; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >  
> > > > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > > > 
> > > > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > > > 
> > > > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > > > 
> > > > And I trust you are running an optimized version of PETSc.
> > > > 
> > > > 
> > > > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > > Hi Song,
> > > > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix? 
> > > > <pastedImage.png>
> > > > 
> > > > 
> > > > Thanks
> > > > 
> > > > Yaxiong Chen, 
> > > > Ph.D. Student 
> > > > 
> > > > School of Mechanical Engineering, 3171 
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > > > Sent: Tuesday, January 29, 2019 1:22 PM
> > > > To: Yaxiong Chen
> > > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >  
> > > > I think you would prefer to preallocate the matrix
> > > > 
> > > > uncomment this line 
> > > > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > > 
> > > > 
> > > > 
> > > > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > > > Hello,
> > > > 
> > > > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > > > 
> > > > Thanks
> > > > 
> > > > block
> > > >     PetscErrorCode ierr
> > > >      PetscMPIInt rank,nproc, mystart
> > > >      PetscInt  nelem
> > > >      integer,allocatable  ::info(:)
> > > >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> > > >      integer, allocatable      :: idx(:)
> > > >      PetscScalar, pointer :: xx_v(:)
> > > >   
> > > >      PC               prec
> > > >      PetscScalar ::   val
> > > >      Vec              xvec,bvec,uvec
> > > >      Mat              Amat
> > > >      KSP              ksp
> > > >      PetscViewer viewer
> > > >      PetscInt  geq,j,k,ne,M,Istart,Iend
> > > >      PetscBool  flg
> > > >      KSPConvergedReason reason
> > > >      Vec         dummyVec, dummyVecs(1)
> > > >      MatNullSpace nullspace
> > > > 
> > > >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > > > 
> > > >      if (ierr .ne. 0) then
> > > >      print*,'Unable to initialize PETSc'
> > > >      stop
> > > >      endif
> > > >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> > > >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > > >      mystart=rank+1
> > > > 
> > > >  !   Parameter set
> > > >      info=ptsystem%getInitInfo()
> > > >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain) 
> > > >      print*,'nelem',nelem
> > > >  !-------------------------------------
> > > >  !  Create Matrix
> > > >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> > > >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> > > >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> > > >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > >      call MatSetFromOptions( Amat, ierr )
> > > >      call MatSetUp( Amat, ierr )
> > > >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> > > >  
> > > >       xvec = tVec(0)
> > > >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> > > >       call VecSetFromOptions( xvec, ierr )
> > > >       call VecDuplicate( xvec, bvec, ierr )
> > > >       call VecDuplicate( xvec, uvec, ierr )
> > > > 
> > > >    t1 = MPI_WTime(); 
> > > > 
> > > >       do i=mystart,nelem,nproc
> > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > >             ne=size(idx)
> > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> > > >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> > > >       end do
> > > > 
> > > >       nelem   = info(Info_EleRHSNum)
> > > >       mystart = rank+1
> > > > 
> > > >       do i = mystart, nelem, nproc  
> > > >         call ptSystem%getElementalRHS(i, RHSe, idx)
> > > >         print*,'idx',idx
> > > >         ne=size(idx)
> > > >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> > > >       end do 
> > > >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> > > >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> > > >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> > > >      t2 = MPI_WTime(); 
> > > >       print*,'assembeling time',t2-t1
> > > >       ! Solve
> > > >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > > > !  Set operators. Here the matrix that defines the linear system
> > > > !  also serves as the preconditioning matrix.
> > > >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> > > >       call KSPSetFromOptions(ksp,ierr)
> > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > !                      Solve the linear system
> > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > >     call KSPSetType(ksp,KSPCG,ierr)
> > > >     call KSPGetPC(ksp,prec,ierr)
> > > >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> > > >     call PCSetType(prec,PCJACOBI,ierr)
> > > >      call KSPSolve(ksp,bvec,xvec,ierr)
> > > >      call PetscFinalize(ierr)
> > > > 
> > > > end block
> > > > 
> > > > 
> > > > Yaxiong Chen, 
> > > > Ph.D. Student 
> > > > 
> > > > School of Mechanical Engineering, 3171 
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907


From chen2018 at purdue.edu  Tue Feb  5 22:48:31 2019
From: chen2018 at purdue.edu (Yaxiong Chen)
Date: Wed, 6 Feb 2019 04:48:31 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <8A05158C-E6BF-4663-9188-60BE3CD7FEB2@mcs.anl.gov>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>
	<DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>
	<DM5PR22MB026540243541443935ACA7A1A86F0@DM5PR22MB0265.namprd22.prod.outlook.com>,
	<8A05158C-E6BF-4663-9188-60BE3CD7FEB2@mcs.anl.gov>
Message-ID: <DM5PR22MB02652AA490BEA2546A7D4BBFA86F0@DM5PR22MB0265.namprd22.prod.outlook.com>

Since mumps and scalapack are already installed on my computer, I only ran ./configure with --download-superlu_dist .

After everything is done, I received the error:

dyld: lazy symbol binding failed: Symbol not found: _MatSolverTypeRegister_SuperLU_DIST

  Referenced from: /Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/lib/libpetsc.3.9.dylib

  Expected in: flat namespace


dyld: Symbol not found: _MatSolverTypeRegister_SuperLU_DIST

  Referenced from: /Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/lib/libpetsc.3.9.dylib

  Expected in: flat namespace


[0]PETSC ERROR: ------------------------------------------------------------------------

[0]PETSC ERROR: Caught signal number 5 TRAP

[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger

[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors

[0]PETSC ERROR: likely location of problem given in stack below

[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------

[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,

[0]PETSC ERROR:       INSTEAD the line number of the start of the function

[0]PETSC ERROR:       is given.

[0]PETSC ERROR: [0] MatInitializePackage line 150 /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/dlregismat.c

[0]PETSC ERROR: [0] MatCreate line 83 /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/utils/gcreate.c

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------

[0]PETSC ERROR: Signal received


________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Tuesday, February 5, 2019 10:58 PM
To: Yaxiong Chen
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] PETSC matrix assembling super slow


  Run ./configure with --download-superlu_dist --download-mumps --download-scalapack and the you can use either -pc_factor_mat_solver_type superlu_dist or -pc_factor_mat_solver_type mumps

  Good luck


> On Feb 5, 2019, at 9:29 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
>
> > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
>
> <<   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly <<the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your <<problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
>
>  << Barry
>
>
> It seems I can only work with Jacobi pre-conditioner. When I try LU or Cholesky ,I got the error :
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.4, Sep, 11, 2018
> [0]PETSC ERROR: ./optimal_mechanical_part.hdc on a arch-darwin-c-debug named hidac.ecn.purdue.edu by chen2018 Tue Feb  5 22:19:08 2019
> [0]PETSC ERROR: Configure options
> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_Cholesky() line 86 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/impls/factor/cholesky/cholesky.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 381 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 612 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
>  solve time   3.8589999999985025E-003
>  reason           0
>  It asks me to download the package. Do I want to run the command in the folder of PETSC? But I am not sure what the name of the package should be . I tried with ./configure --download-<PCLU> But it does not work.
>
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, February 5, 2019 1:18 PM
> To: Yaxiong Chen
> Cc: PETSc users list
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>
>
>
> > On Feb 5, 2019, at 10:32 AM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> >
> > Thanks Barry,
> >      I will explore how to partition for parallel computation later. But now I still have some confusion on the sequential operation.
> > I compared PETSC and Mumps. In both case, the subroutine for generating elemental matrix is very similar. However, the difference is in the following step?
> >    call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > In this step ,each element cost about 3~4 e-2 second.
> >
> > However, when I use mumps, I use the following code:
> >    preA(Aptr+1:Aptr+n*(n+1)/2) = pack(Ae, mask(1:n,1:n))
> >    Aptr = Aptr +n*(n+1)/2
> >    if (allocated(auxRHSe)) preRHS(idx) = preRHS(idx)+auxRHSe
> >  It just cost 10e-6~10e-5 second. For a 10000*10000 matrix, the assembling time for using PETSC is 300s while it cost 60s when using Mumps.
> > For a 10000*10000 system,the  Is there any way I can make it faster?
>
>   As we keep saying the slowness of the matrix assembly is due to incorrect preallocation. Once you have the preallocation correct the speed should increase dramatically.
>
> > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
>
>    It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
>
>   Barry
>
>
> >
> > Thanks
> >
> > Yaxiong
> >
> >
> >
> >
> >
> > Yaxiong Chen,
> > Ph.D. Student
> >
> > School of Mechanical Engineering, 3171
> > 585 Purdue Mall
> > West Lafayette, IN 47907
> >
> >
> >
> >
> >
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Monday, February 4, 2019 10:42 PM
> > To: Yaxiong Chen
> > Cc: PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >
> >
> >
> > > On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > >
> > >   Hi Barry,
> > >
> > >  !===================================================
> > >       mystart =rank +1                                    ! rank starts from 0
> > >       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process
> > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
> > >             ne=size(idx)
> > >             idx=idx-1           !-1 since PETSC index starts from zero
> > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
> > >       end do
> > > !===================================================
> > >   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.
> >
> >    This is very bad strategy because there is no data locality.
> >
> > > Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?
> >
> >   What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.
> >
> >    Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.
> >
> >    Barry
> >
> > >
> > > Thanks
> > >
> > > Yaxiong
> > >
> > >
> > > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > > Sent: Monday, February 4, 2019 5:21 PM
> > > To: Yaxiong Chen
> > > Cc: Mark Adams; PETSc users list
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >
> > >
> > >
> > > > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > >
> > > > Hi Mark,
> > > >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> > > >       do i=mystart,nelem,nproc
> > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > >             ne=size(idx)
> > > >             idx=idx-1           !-1 since PETSC index starts from zero
> > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > > >       end do
> > > > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> > > >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > >
> > > > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> > >
> > >
> > >    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> > >
> > >
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> > > >  aseembel time   282.54871800000001
> > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > > > [0] PCSetUp(): Setting up PC for first time
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > >
> > > > However, when I use the parallel mode,I got the following information:
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> > >
> > >    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> > >
> > >   Barry
> > >
> > >
> > >
> > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > >
> > > > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > > Is there anything I am doing wrong?
> > > > Thanks
> > > >
> > > > Yaxiong Chen,
> > > > Ph.D. Student
> > > >
> > > > School of Mechanical Engineering, 3171
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Mark Adams <mfadams at lbl.gov>
> > > > Sent: Tuesday, January 29, 2019 10:02 PM
> > > > To: Yaxiong Chen; PETSc users list
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >
> > > > Optimized is a configuration flag not a versions.
> > > >
> > > > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > > >
> > > > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > > >
> > > > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > > Thanks Mark,
> > > >     I use PETSC 3.9.4, is this the optimized version you called?
> > > >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > > >
> > > > Yaxiong Chen,
> > > > Ph.D. Student
> > > >
> > > > School of Mechanical Engineering, 3171
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Mark Adams <mfadams at lbl.gov>
> > > > Sent: Tuesday, January 29, 2019 8:25 PM
> > > > To: Yaxiong Chen
> > > > Cc: Song Gao; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >
> > > > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > > >
> > > > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > > >
> > > > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > > >
> > > > And I trust you are running an optimized version of PETSc.
> > > >
> > > >
> > > > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > > Hi Song,
> > > > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix?
> > > > <pastedImage.png>
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Yaxiong Chen,
> > > > Ph.D. Student
> > > >
> > > > School of Mechanical Engineering, 3171
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > > > Sent: Tuesday, January 29, 2019 1:22 PM
> > > > To: Yaxiong Chen
> > > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >
> > > > I think you would prefer to preallocate the matrix
> > > >
> > > > uncomment this line
> > > > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > >
> > > >
> > > >
> > > > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > > > Hello,
> > > >
> > > > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > > >
> > > > Thanks
> > > >
> > > > block
> > > >     PetscErrorCode ierr
> > > >      PetscMPIInt rank,nproc, mystart
> > > >      PetscInt  nelem
> > > >      integer,allocatable  ::info(:)
> > > >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> > > >      integer, allocatable      :: idx(:)
> > > >      PetscScalar, pointer :: xx_v(:)
> > > >
> > > >      PC               prec
> > > >      PetscScalar ::   val
> > > >      Vec              xvec,bvec,uvec
> > > >      Mat              Amat
> > > >      KSP              ksp
> > > >      PetscViewer viewer
> > > >      PetscInt  geq,j,k,ne,M,Istart,Iend
> > > >      PetscBool  flg
> > > >      KSPConvergedReason reason
> > > >      Vec         dummyVec, dummyVecs(1)
> > > >      MatNullSpace nullspace
> > > >
> > > >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > > >
> > > >      if (ierr .ne. 0) then
> > > >      print*,'Unable to initialize PETSc'
> > > >      stop
> > > >      endif
> > > >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> > > >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > > >      mystart=rank+1
> > > >
> > > >  !   Parameter set
> > > >      info=ptsystem%getInitInfo()
> > > >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain)
> > > >      print*,'nelem',nelem
> > > >  !-------------------------------------
> > > >  !  Create Matrix
> > > >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> > > >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> > > >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> > > >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > >      call MatSetFromOptions( Amat, ierr )
> > > >      call MatSetUp( Amat, ierr )
> > > >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> > > >
> > > >       xvec = tVec(0)
> > > >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> > > >       call VecSetFromOptions( xvec, ierr )
> > > >       call VecDuplicate( xvec, bvec, ierr )
> > > >       call VecDuplicate( xvec, uvec, ierr )
> > > >
> > > >    t1 = MPI_WTime();
> > > >
> > > >       do i=mystart,nelem,nproc
> > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > >             ne=size(idx)
> > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> > > >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> > > >       end do
> > > >
> > > >       nelem   = info(Info_EleRHSNum)
> > > >       mystart = rank+1
> > > >
> > > >       do i = mystart, nelem, nproc
> > > >         call ptSystem%getElementalRHS(i, RHSe, idx)
> > > >         print*,'idx',idx
> > > >         ne=size(idx)
> > > >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> > > >       end do
> > > >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> > > >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> > > >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> > > >      t2 = MPI_WTime();
> > > >       print*,'assembeling time',t2-t1
> > > >       ! Solve
> > > >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > > > !  Set operators. Here the matrix that defines the linear system
> > > > !  also serves as the preconditioning matrix.
> > > >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> > > >       call KSPSetFromOptions(ksp,ierr)
> > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > !                      Solve the linear system
> > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > >     call KSPSetType(ksp,KSPCG,ierr)
> > > >     call KSPGetPC(ksp,prec,ierr)
> > > >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> > > >     call PCSetType(prec,PCJACOBI,ierr)
> > > >      call KSPSolve(ksp,bvec,xvec,ierr)
> > > >      call PetscFinalize(ierr)
> > > >
> > > > end block
> > > >
> > > >
> > > > Yaxiong Chen,
> > > > Ph.D. Student
> > > >
> > > > School of Mechanical Engineering, 3171
> > > > 585 Purdue Mall
> > > > West Lafayette, IN 47907

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/b3f5cbf5/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Feb  5 23:14:29 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 6 Feb 2019 05:14:29 +0000
Subject: [petsc-users] PETSC matrix assembling super slow
In-Reply-To: <DM5PR22MB02652AA490BEA2546A7D4BBFA86F0@DM5PR22MB0265.namprd22.prod.outlook.com>
References: <DM5PR22MB0265F47E39C683B17018E8ADA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CAJitgPW51fDB0m6JD6wZtnsY64TCvv4Lswmgy-d4JWnMcfeSWw@mail.gmail.com>
	<DM5PR22MB0265D812AC3D2444B1650B9BA8970@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh71a2PPJdvNG2hAHrwD-xjapze6seQBXsrdpxGG4bqhMA@mail.gmail.com>
	<DM5PR22MB02655D80D5ACF75F832188B8A8900@DM5PR22MB0265.namprd22.prod.outlook.com>
	<CADOhEh4drGpPWnC8bVuXomn-pNm1s6xV_YzqcrjGS_KM9-JHnw@mail.gmail.com>
	<DM5PR22MB026520F6F40714C29109EF9DA86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<B8E36B56-2277-456D-9A4B-9736F05F5452@anl.gov>
	<DM5PR22MB0265C9106626384132314F54A86D0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<3B4EEBF1-F549-4AE9-9663-5E854EDC00C8@mcs.anl.gov>
	<DM5PR22MB026565FC4A8D12A73EE3F5F3A86E0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<C6BBBD69-396D-48D2-A9AF-B355B9F6A702@mcs.anl.gov>
	<DM5PR22MB026540243541443935ACA7A1A86F0@DM5PR22MB0265.namprd22.prod.outlook.com>
	<8A05158C-E6BF-4663-9188-60BE3CD7FEB2@mcs.anl.gov>
	<DM5PR22MB02652AA490BEA2546A7D4BBFA86F0@DM5PR22MB0265.namprd22.prod.outlook.com>
Message-ID: <12220FA1-E33D-4728-AC41-52E7EDAF7FF7@mcs.anl.gov>


  Send configure.log and make.log to petsc-maint at mcs.anl.gov

  Barry


> On Feb 5, 2019, at 10:48 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> 
> Since mumps and scalapack are already installed on my computer, I only ran ./configure with --download-superlu_dist . 
> After everything is done, I received the error: 
> dyld: lazy symbol binding failed: Symbol not found: _MatSolverTypeRegister_SuperLU_DIST
>   Referenced from: /Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/lib/libpetsc.3.9.dylib
>   Expected in: flat namespace
> 
> dyld: Symbol not found: _MatSolverTypeRegister_SuperLU_DIST
>   Referenced from: /Users/yaxiong/Downloads/petsc-3.9.4/arch-darwin-c-debug/lib/libpetsc.3.9.dylib
>   Expected in: flat namespace
> 
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 5 TRAP
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatInitializePackage line 150 /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/dlregismat.c
> [0]PETSC ERROR: [0] MatCreate line 83 /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/utils/gcreate.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> 
> 
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Tuesday, February 5, 2019 10:58 PM
> To: Yaxiong Chen
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PETSC matrix assembling super slow
>  
> 
>   Run ./configure with --download-superlu_dist --download-mumps --download-scalapack and the you can use either -pc_factor_mat_solver_type superlu_dist or -pc_factor_mat_solver_type mumps
> 
>   Good luck
> 
> 
> > On Feb 5, 2019, at 9:29 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > 
> > > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
> > 
> > <<   It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly <<the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your <<problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
> > 
> >  << Barry
> > 
> > 
> > It seems I can only work with Jacobi pre-conditioner. When I try LU or Cholesky ,I got the error :
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
> > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.9.4, Sep, 11, 2018 
> > [0]PETSC ERROR: ./optimal_mechanical_part.hdc on a arch-darwin-c-debug named hidac.ecn.purdue.edu by chen2018 Tue Feb  5 22:19:08 2019
> > [0]PETSC ERROR: Configure options 
> > [0]PETSC ERROR: #1 MatGetFactor() line 4328 in /Users/yaxiong/Downloads/petsc-3.9.4/src/mat/interface/matrix.c
> > [0]PETSC ERROR: #2 PCSetUp_Cholesky() line 86 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/impls/factor/cholesky/cholesky.c
> > [0]PETSC ERROR: #3 PCSetUp() line 923 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: #4 KSPSetUp() line 381 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: #5 KSPSolve() line 612 in /Users/yaxiong/Downloads/petsc-3.9.4/src/ksp/ksp/interface/itfunc.c
> >  solve time   3.8589999999985025E-003
> >  reason           0
> >  It asks me to download the package. Do I want to run the command in the folder of PETSC? But I am not sure what the name of the package should be . I tried with ./configure --download-<PCLU> But it does not work.
> > 
> > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > Sent: Tuesday, February 5, 2019 1:18 PM
> > To: Yaxiong Chen
> > Cc: PETSc users list
> > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> >  
> > 
> > 
> > > On Feb 5, 2019, at 10:32 AM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > 
> > > Thanks Barry,
> > >      I will explore how to partition for parallel computation later. But now I still have some confusion on the sequential operation.
> > > I compared PETSC and Mumps. In both case, the subroutine for generating elemental matrix is very similar. However, the difference is in the following step?
> > >    call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > > In this step ,each element cost about 3~4 e-2 second.
> > > 
> > > However, when I use mumps, I use the following code: 
> > >    preA(Aptr+1:Aptr+n*(n+1)/2) = pack(Ae, mask(1:n,1:n))
> > >    Aptr = Aptr +n*(n+1)/2
> > >    if (allocated(auxRHSe)) preRHS(idx) = preRHS(idx)+auxRHSe
> > >  It just cost 10e-6~10e-5 second. For a 10000*10000 matrix, the assembling time for using PETSC is 300s while it cost 60s when using Mumps.
> > > For a 10000*10000 system,the  Is there any way I can make it faster? 
> > 
> >   As we keep saying the slowness of the matrix assembly is due to incorrect preallocation. Once you have the preallocation correct the speed should increase dramatically.
> > 
> > > Also, I found the solving time is also shorter when I use the direct solver(0.432s  vs 4.332 s ). Is this due the small scale of the system? When I have a very large (e.g., 100000*100000 ) system, can I expect iterative solver being faster?
> > 
> >    It sounds like the default preconditioner is not working well on your problem. First run with -ksp_monitor -ksp_converged_reason to get an idea of how quickly the iterative solver is converging. If the number of iterations is high you are going to need to change the preconditioner to get one that converges well for your problem. What mathematical models is your code implementing, this will help in determining what type of preconditioner to use.
> > 
> >   Barry
> > 
> > 
> > > 
> > > Thanks
> > > 
> > > Yaxiong
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Yaxiong Chen, 
> > > Ph.D. Student 
> > > 
> > > School of Mechanical Engineering, 3171 
> > > 585 Purdue Mall
> > > West Lafayette, IN 47907
> > > 
> > > 
> > > 
> > > 
> > > 
> > > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > > Sent: Monday, February 4, 2019 10:42 PM
> > > To: Yaxiong Chen
> > > Cc: PETSc users list
> > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > >  
> > > 
> > > 
> > > > On Feb 4, 2019, at 4:41 PM, Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > > 
> > > >   Hi Barry,
> > > > 
> > > >  !===================================================
> > > >       mystart =rank +1                                    ! rank starts from 0
> > > >       do i=mystart,nelem,nproc                    ! nelem: total number of elements  ; nproc :number of process 
> > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)    ! Generate elemental matrix Ae and corresponding global Idx
> > > >             ne=size(idx)            
> > > >             idx=idx-1           !-1 since PETSC index starts from zero 
> > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)  
> > > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr) ! Add elemental RHS to global RHS
> > > >       end do
> > > > !===================================================
> > > >   Maybe this is where I am wrong. The way I use MPI is to let each core generate the elemental matrix in turn.
> > > 
> > >    This is very bad strategy because there is no data locality. 
> > > 
> > > > Which means I have one global matrix on each core and finally add them together. My case is similar to typical finite element method. But the problem is the Index is not continuous, in this case I don't know how I can partition the global matrix. Do you have any suggestions or do you have any template which can show me how finite element method use PETSC?
> > > 
> > >   What you need to do is partition the elements across the processors (so that the each process has a contiguous subdomain of elements), The each process computes the element stiffness for "its elements". There really isn't a single PETSc example that manages this all directly for finite elements because that is a rather involved process to do it so as to get good performance.
> > > 
> > >    Depending on how specialized your problem needs to be you might consider using one of the packages libMesh, MOOSE or deal.ii to manage the elements and element computations (they all use PETSc for the algebraic solvers) instead of doing it yourself; it is an involved process to do it all your self.
> > > 
> > >    Barry
> > > 
> > > > 
> > > > Thanks
> > > > 
> > > > Yaxiong
> > > > 
> > > > 
> > > > From: Smith, Barry F. <bsmith at mcs.anl.gov>
> > > > Sent: Monday, February 4, 2019 5:21 PM
> > > > To: Yaxiong Chen
> > > > Cc: Mark Adams; PETSc users list
> > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > >  
> > > > 
> > > > 
> > > > > On Feb 4, 2019, at 3:17 PM, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > > > 
> > > > > Hi Mark,
> > > > >      Will the parameter MatMPIAIJSetPreallocation in influence the following part
> > > > >       do i=mystart,nelem,nproc
> > > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > > >             ne=size(idx)
> > > > >             idx=idx-1           !-1 since PETSC index starts from zero 
> > > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx,auxRHSe,ADD_VALUES,ierr)
> > > > >             call MatSetValues(Amat,ne,idx,ne,idx,Ae,ADD_VALUES,ierr)
> > > > >       end do
> > > > > I found this part will impede my assembling process. In my case ,the total DOF is 20800. I estimated the upper bound  of number of nonzero entries in each row as 594. So I set f9 to be 20206 and f6 to be 10103 in the following command:
> > > > >  call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > > > 
> > > > > Running the code in sequential mode with -info I got the following information.And I can get the result even thought the assembling process is kind of slow(several times slower than using Mumps).
> > > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 0 entries,uses 0 mallocs.
> > > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 20800, block size 1; storage space: 13847 unneeded, 1030038 used
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 62659
> > > > 
> > > > 
> > > >    Something is very wrong. The number of mallocs should be zero if you have the correct preallocation. Are you calling MatZeroEntries() or some other Mat routine before you call MatSetValues() ?
> > > > 
> > > > 
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 70
> > > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 20800) < 0.6. Do not use CompressedRow routines.
> > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462889472 140509341618528
> > > > > [0] VecScatterCreate_Seq(): Special case: sequential copy
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 20800 X 0, block size 1; storage space: 104000 unneeded, 0 used
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0
> > > > > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0
> > > > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 20800)/(num_localrows 20800) > 0.6. Use CompressedRow routines.
> > > > >  aseembel time   282.54871800000001     
> > > > > [0] PetscCommDuplicate(): Using internal PETSc communicator 4462888960 140509341442256
> > > > > [0] PCSetUp(): Setting up PC for first time
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged
> > > > > 
> > > > > However, when I use the parallel mode,I got the following information:
> > > > > [0] MatAssemblyBegin_MPIBAIJ(): Stash has 707965 entries,uses 6 mallocs.
> > > > 
> > > >    You have a lot of off-process MatSetValues(). That is one process is generating a lot of matrix entries that belong on another process. This is not desirable. Ideally each process will generate the matrix entries that belong to that process so only a few matrix entries need to be transported to another process. How are you partitioning the mesh and how are you deciding which process computes which which entries in the matrix? All of this may be need to be revisited.
> > > > 
> > > >   Barry
> > > > 
> > > > 
> > > > 
> > > > > [0] MatAssemblyBegin_MPIBAIJ(): Block-Stash has 0 entries, uses 0 mallocs.
> > > > > 
> > > > > it seems it never went to  call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > > > Is there anything I am doing wrong?
> > > > > Thanks
> > > > > 
> > > > > Yaxiong Chen, 
> > > > > Ph.D. Student 
> > > > > 
> > > > > School of Mechanical Engineering, 3171 
> > > > > 585 Purdue Mall
> > > > > West Lafayette, IN 47907
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > From: Mark Adams <mfadams at lbl.gov>
> > > > > Sent: Tuesday, January 29, 2019 10:02 PM
> > > > > To: Yaxiong Chen; PETSc users list
> > > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > > >  
> > > > > Optimized is a configuration flag not a versions.
> > > > > 
> > > > > You need to figure out your number of non-zeros per row of you global matrix, or a bound on it, and supply that in MatMPIAIJSetPreallocation. Otherwise it has to allocate and copy memory often.
> > > > > 
> > > > > You could increase your f9 on a serial run and see what runs best and then  move to parallel with a value in f6 of about 1/2 of f9.
> > > > > 
> > > > > On Tue, Jan 29, 2019 at 9:13 PM Yaxiong Chen <chen2018 at purdue.edu> wrote:
> > > > > Thanks Mark, 
> > > > >     I use PETSC 3.9.4, is this the optimized version you called?
> > > > >     Actually f9 and f6 are from the PETSC example. I don't know how to set the value correctly so I commend them. The size of my elemental matrix may vary. For 2D problem, the size of elemental matrix can be 24*24 or 32*32 or some other sizes. And the index is not continuous. In this case, the elemental matrix may interlace with each other in the global matrix, and I may have thousands of elemental matrix to be assembled. Does the preallocating suitable for this?
> > > > > 
> > > > > Yaxiong Chen, 
> > > > > Ph.D. Student 
> > > > > 
> > > > > School of Mechanical Engineering, 3171 
> > > > > 585 Purdue Mall
> > > > > West Lafayette, IN 47907
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > From: Mark Adams <mfadams at lbl.gov>
> > > > > Sent: Tuesday, January 29, 2019 8:25 PM
> > > > > To: Yaxiong Chen
> > > > > Cc: Song Gao; petsc-users at mcs.anl.gov
> > > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > > >  
> > > > > Slow assembly is often from not preallocating correctly. I am guessing that you are using Q1 element and f9==9, and thus the preallocation should be OK if this is a scalar problem on a regular grid and f6-==6 should be OK for the off processor allocation, if my assumptions are correct.
> > > > > 
> > > > > You can run with -info, which will tell you how many allocation were done in assembly. Make sure that it is small (eg, 0).
> > > > > 
> > > > > I see you use f90 array stuff 'idx-1'. Compilers can sometimes do crazy things with seeming simple code. You could just do this manually if you can find anything else.
> > > > > 
> > > > > And I trust you are running an optimized version of PETSc.
> > > > > 
> > > > > 
> > > > > On Tue, Jan 29, 2019 at 6:22 PM Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > > > Hi Song,
> > > > > I don't quite understand how I can use this command. I don't partition the gloabl matrix. If I add my elemental matrix to the global system it will be like this. And in my parallel part, I use each core to generate the elemental matrix in turn. In this case, I guess each core will be assigned the space for global matrix and finally be assembled.But according to the manual, it seems each core will store a part of the global matrix. Is the local submatrix in the MatMPIAIJSetPreallocation(Mat B,PetscInt d_nz,const PetscInt d_nnz[],PetscInt o_nz,const PetscInto_nnz[])the same as my elemental matrix? 
> > > > > <pastedImage.png>
> > > > > 
> > > > > 
> > > > > Thanks
> > > > > 
> > > > > Yaxiong Chen, 
> > > > > Ph.D. Student 
> > > > > 
> > > > > School of Mechanical Engineering, 3171 
> > > > > 585 Purdue Mall
> > > > > West Lafayette, IN 47907
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > From: Song Gao <song.gao2 at mail.mcgill.ca>
> > > > > Sent: Tuesday, January 29, 2019 1:22 PM
> > > > > To: Yaxiong Chen
> > > > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > > > > Subject: Re: [petsc-users] PETSC matrix assembling super slow
> > > > >  
> > > > > I think you would prefer to preallocate the matrix
> > > > > 
> > > > > uncomment this line 
> > > > > ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > > > 
> > > > > 
> > > > > 
> > > > > Le mar. 29 janv. 2019, ? 12 h 40, Yaxiong Chen via petsc-users <petsc-users at mcs.anl.gov> a ?crit :
> > > > > Hello,
> > > > > 
> > > > > I have a 2D system which is assembled by each elemental matrix.  Ae is my elemental matrix, auxRHSe(:) and RHSe(:) and corresponding right hand side, idx is the global index. My code is as follow, however ,the assembling rate is super slow(Marked red in the code). I am not sure whether the assembling type is right or not. Since for each element, idx are not continuous numbers. Do you have any idea what is the better way to assemble the matrix?
> > > > > 
> > > > > Thanks
> > > > > 
> > > > > block
> > > > >     PetscErrorCode ierr
> > > > >      PetscMPIInt rank,nproc, mystart
> > > > >      PetscInt  nelem
> > > > >      integer,allocatable  ::info(:)
> > > > >      real(wp), allocatable :: Ae(:,:), auxRHSe(:),RHSe(:)
> > > > >      integer, allocatable      :: idx(:)
> > > > >      PetscScalar, pointer :: xx_v(:)
> > > > >   
> > > > >      PC               prec
> > > > >      PetscScalar ::   val
> > > > >      Vec              xvec,bvec,uvec
> > > > >      Mat              Amat
> > > > >      KSP              ksp
> > > > >      PetscViewer viewer
> > > > >      PetscInt  geq,j,k,ne,M,Istart,Iend
> > > > >      PetscBool  flg
> > > > >      KSPConvergedReason reason
> > > > >      Vec         dummyVec, dummyVecs(1)
> > > > >      MatNullSpace nullspace
> > > > > 
> > > > >      call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
> > > > > 
> > > > >      if (ierr .ne. 0) then
> > > > >      print*,'Unable to initialize PETSc'
> > > > >      stop
> > > > >      endif
> > > > >      call MPI_Comm_size(PETSC_COMM_WORLD,nproc,ierr)
> > > > >      call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr)
> > > > >      mystart=rank+1
> > > > > 
> > > > >  !   Parameter set
> > > > >      info=ptsystem%getInitInfo()
> > > > >      nelem = info(Info_EleMatNum)+info(Info_FixedDOFNum)+info(Info_NumConstrain) 
> > > > >      print*,'nelem',nelem
> > > > >  !-------------------------------------
> > > > >  !  Create Matrix
> > > > >      call MatCreate(PETSC_COMM_WORLD,Amat,ierr)
> > > > >      call MatSetSizes( Amat,PETSC_DECIDE, PETSC_DECIDE, info(1), info(1), ierr )
> > > > >      call MatSetType( Amat, MATMPIBAIJ, ierr )
> > > > >      ! call MatMPIAIJSetPreallocation(Amat,f9,PETSC_NULL_INTEGER,f6,PETSC_NULL_INTEGER, ierr)
> > > > >      call MatSetFromOptions( Amat, ierr )
> > > > >      call MatSetUp( Amat, ierr )
> > > > >      call MatGetOwnershipRange( Amat, Istart, Iend, ierr )
> > > > >  
> > > > >       xvec = tVec(0)
> > > > >       call MatCreateVecs( Amat, PETSC_NULL_VEC, xvec, ierr )
> > > > >       call VecSetFromOptions( xvec, ierr )
> > > > >       call VecDuplicate( xvec, bvec, ierr )
> > > > >       call VecDuplicate( xvec, uvec, ierr )
> > > > > 
> > > > >    t1 = MPI_WTime(); 
> > > > > 
> > > > >       do i=mystart,nelem,nproc
> > > > >             call ptSystem%getElementalMAT(i, Ae, auxRHSe, idx)
> > > > >             ne=size(idx)
> > > > >             if (allocated(auxRHSe))  call VecSetValues(bvec,ne,idx-1,auxRHSe,ADD_VALUES,ierr)
> > > > >              call MatSetValues(Amat,ne,idx-1,ne,idx-1,Ae,ADD_VALUES,ierr)
> > > > >       end do
> > > > > 
> > > > >       nelem   = info(Info_EleRHSNum)
> > > > >       mystart = rank+1
> > > > > 
> > > > >       do i = mystart, nelem, nproc  
> > > > >         call ptSystem%getElementalRHS(i, RHSe, idx)
> > > > >         print*,'idx',idx
> > > > >         ne=size(idx)
> > > > >         if (allocated(RHSe))  call VecSetValues(bvec,ne,idx-1,RHSe,ADD_VALUES,ierr)
> > > > >       end do 
> > > > >       call MatAssemblyBegin(Amat,MAT_FINAL_ASSEMBLY,ierr)
> > > > >       call MatAssemblyEnd(Amat,MAT_FINAL_ASSEMBLY,ierr)                          !  this part is slow, the for loop above is done but here it may get stuck
> > > > >       call VecAssemblyBegin(bvec,ierr)                                                                    ! For a 2500 DOF system, assembling only takes over 2 seconds
> > > > >       call VecAssemblyEnd(bvec,ierr)                                                                      ! But for a 10000 DOF system , it gets stuck
> > > > >      t2 = MPI_WTime(); 
> > > > >       print*,'assembeling time',t2-t1
> > > > >       ! Solve
> > > > >       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> > > > > !  Set operators. Here the matrix that defines the linear system
> > > > > !  also serves as the preconditioning matrix.
> > > > >       call KSPSetOperators(ksp,Amat,Amat,ierr)
> > > > >       call KSPSetFromOptions(ksp,ierr)
> > > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > > !                      Solve the linear system
> > > > > ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > > > >     call KSPSetType(ksp,KSPCG,ierr)
> > > > >     call KSPGetPC(ksp,prec,ierr)
> > > > >    ! call KSPSetPCSide(ksp,PC_SYMMETRIC,ierr)
> > > > >     call PCSetType(prec,PCJACOBI,ierr)
> > > > >      call KSPSolve(ksp,bvec,xvec,ierr)
> > > > >      call PetscFinalize(ierr)
> > > > > 
> > > > > end block
> > > > > 
> > > > > 
> > > > > Yaxiong Chen, 
> > > > > Ph.D. Student 
> > > > > 
> > > > > School of Mechanical Engineering, 3171 
> > > > > 585 Purdue Mall
> > > > > West Lafayette, IN 47907


From knepley at gmail.com  Wed Feb  6 05:32:40 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 Feb 2019 06:32:40 -0500
Subject: [petsc-users] reading petsc binary files.
In-Reply-To: <64428CEE-5452-42C8-B4D1-84FD2E2EAEA2@anl.gov>
References: <0dff9cee1c044728974dcf4462f39597@lhsc.on.ca>
	<64428CEE-5452-42C8-B4D1-84FD2E2EAEA2@anl.gov>
Message-ID: <CAMYG4GmE9KUsqM_hJFwzuiNJNZa0HJOfMdG3jwMA8mxYWwG9kA@mail.gmail.com>

On Tue, Feb 5, 2019 at 6:36 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> > On Feb 5, 2019, at 5:06 PM, Sanjay Kharche via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> >
> > Hi
> >
> > I use two mpi clusters (cluster 1 and 2). Whereas the petsc binary files
> I generate can be read on cluster 1, I get errors doing so on cluster 2. I
> also output vts files corresponding to each binary file output, and it
> appears that both clusters do produce meaningful results. I use ver 3.7.4
> on cluster 1, and 3.7.5 on 2. Also, the same simulation produces binary
> files of slightly different sizes on the two clusters.
>
>    This is truly strange. PETSc has always used the exact same format for
> its binary files and automatically handles any byte-swapping that may be
> need to always have a consistent representation on the disk. Also a
> difference in the file size is strange. In over 20 years we've never had an
> issue with unreadable binary files.
>
>   Can cluster 1 read the binary files from cluster 1 and cluster 2 read
> the binary files from cluster 2. Is it just cluster 2 cannot read the files
> from cluster 1?
>
>   Could the file be changed somehow as it is copied between the clusters?
>
>   Is this difference in size reproducible if you delete the files and
> create them again?
>
>   Is one of the clusters by any chance a Microsoft Windows cluster?
>

Is one configured with 64-bit integers and the other not?

  Thanks,

     Matt


>    Barry
>
> > Can you comment on what I need to do to be able to read binary files on
> cluster 2.
> >
> > thanks
> > Sanjay
> >
> > Code snippet from parallel petsc code that does output:
> >
> > if(time_int % (int)(1.0/DELTAT) == 0){ // a smaller time step and more
> files outputted makes the CV estimate better.
> > sprintf(str,"my_2d%d.vts", file_Counter); // this confirms that
> simulation does something meaningful.
> > PetscViewer viewer;  PetscViewerCreate(PETSC_COMM_WORLD, &viewer);
> PetscViewerSetType(viewer, PETSCVIEWERVTK);
> > PetscViewerFileSetName(viewer, str); VecView(u, viewer);
> PetscViewerDestroy(&viewer);
> >                 sprintf(str,"my_3d%d.bin",(int)file_Counter);
> > PetscViewer viewer2;
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE,&viewer2);
> > VecView(u,viewer2);
> > PetscViewerDestroy(&viewer2);
> > file_Counter++;
> > }
> >
> >
> > How I am trying to read it (typically serial code with binary called
> ecg):
> >
> >   // inputs are Petsc binary files, create and destroy viewer at each
> file for simplicity.
> > PetscViewer viewer_in;
> > sprintf(str,"my_3d%d.bin",file_Counter);
> > PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_READ,&viewer_in);
> > VecLoad(u,viewer_in);
> > PetscViewerDestroy(&viewer_in);
> >
> >
> > Errors I got when I ran ecg:
> >
> > login3 endo]$ ./ecg
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Invalid argument
> > [0]PETSC ERROR: Not a vector next in file
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017
> > [0]PETSC ERROR: ./ecg on a arch-linux2-c-opt named gra-login3 by
> kharches Tue Feb  5 17:43:36 2019
> > [0]PETSC ERROR: Configure options --prefix=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/petsc/3.7.5
> --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl
> --with-hdf5=1 --with-hdf5-dir=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18
> --download-hypre=1 --download-metis=1 --download-triangle=1
> --download-ptscotch=1 --download-superlu_dist=1 --download-ml=1
> --download-superlu=1 --download-prometheus=1 --download-mumps=1
> --download-parmetis=1 --download-suitesparse=1 --download-mumps-shared=0
> --download-ptscotch-shared=0 --download-superlu-shared=0
> --download-superlu_dist-shared=0 --download-parmetis-shared=0
> --download-metis-shared=0 --download-ml-shared=0
> --download-suitesparse-shared=0 --download-hypre-shared=0
> --download-prometheus-shared=0 --with-cc=mpicc --with-cxx=mpicxx
> --with-c++-support --with-fc=mpifort --CFLAGS="-O2 -xCore-AVX2 -ftz
> -fp-speculation=safe -fp-model source -fPIC" --CXXFLAGS="-O2 -xCore-AVX2
> -ftz -fp-speculation=safe -fp-model source -fPIC" --FFLAGS="-O2 -xCore-AVX2
> -ftz -fp-speculation=safe -fp-model source -fPIC" --with-gnu-compilers=0
> --with-mpi=1 --with-build-step-np=8 --with-shared-libraries=1
> --with-debugging=0 --with-pic=1 --with-x=0 --with-windows-graphics=0
> --with-scalapack=1 --with-scalapack-include=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/include
> --with-scalapack-lib="[/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_scalapack_lp64.a,libmkl_blacs_openmpi_lp64.a,libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]"
> --with-blas-lapack-lib="[/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a]"
> --with-hdf5=1 --with-hdf5-dir=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18
> --with-fftw=1 --with-fftw-dir=/cvmfs/
> soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/fftw-mpi/3.3.6
> > [0]PETSC ERROR: #1 PetscViewerBinaryReadVecHeader_Private() line 28 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> > [0]PETSC ERROR: #2 VecLoad_Binary() line 90 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> > [0]PETSC ERROR: #3 VecLoad_Default() line 413 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> > [0]PETSC ERROR: #4 VecLoad() line 975 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
> > [0]PETSC ERROR: #5 VecLoad_Binary_DA() line 931 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> > [0]PETSC ERROR: #6 VecLoad_Default_DA() line 964 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> > [0]PETSC ERROR: #7 VecLoad() line 975 in
> /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
> >
> > This email is directed in confidence solely to the person named above
> and may contain confidential, privileged or personal health information.
> Please be aware that this email may also be released to members of the
> public under Ontario's Freedom of Information and Protection of Privacy Act
> if required. Review, distribution, or disclosure of this email by anyone
> other than the person(s) for whom it was originally intended is strictly
> prohibited. If you are not an intended recipient, please notify the sender
> immediately via a return email and destroy all copies of the original
> message. Thank you for your cooperation.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/eba62ef5/attachment-0001.html>

From Sanjay.Kharche at lhsc.on.ca  Wed Feb  6 09:49:54 2019
From: Sanjay.Kharche at lhsc.on.ca (Sanjay Kharche)
Date: Wed, 6 Feb 2019 15:49:54 +0000
Subject: [petsc-users] reading petsc binary files.
In-Reply-To: <CAMYG4GmE9KUsqM_hJFwzuiNJNZa0HJOfMdG3jwMA8mxYWwG9kA@mail.gmail.com>
References: <0dff9cee1c044728974dcf4462f39597@lhsc.on.ca>
	<64428CEE-5452-42C8-B4D1-84FD2E2EAEA2@anl.gov>,
	<CAMYG4GmE9KUsqM_hJFwzuiNJNZa0HJOfMdG3jwMA8mxYWwG9kA@mail.gmail.com>
Message-ID: <f5afb4f0f23d44abacf734b4c9630522@lhsc.on.ca>


Hi Matt, Barry

Thanks for rapidly replying.

Both clusters are 64 bit, neither is MS Windows (wouldnt know how to use a windows machine), nor is my file transfer on windows. Just did some test runs with various combinations. It appears that it was a one off where bin files got corrupted during sftp/scp (I forgot which I used, always on Fedora cluster 1, red hat enterprise cluster 2). I have other queries, but I will make another posting for that tonight/tomorrow.

thanks for pointing out that sftp/scp may need (but not always) to be in binary mode unless I want a 4 bit word attached to the transferred file.

Sanjay

________________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Wednesday, February 6, 2019 6:32:40 AM
To: Smith, Barry F.
Cc: Sanjay Kharche; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] reading petsc binary files.

On Tue, Feb 5, 2019 at 6:36 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> On Feb 5, 2019, at 5:06 PM, Sanjay Kharche via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
>
> Hi
>
> I use two mpi clusters (cluster 1 and 2). Whereas the petsc binary files I generate can be read on cluster 1, I get errors doing so on cluster 2. I also output vts files corresponding to each binary file output, and it appears that both clusters do produce meaningful results. I use ver 3.7.4 on cluster 1, and 3.7.5 on 2. Also, the same simulation produces binary files of slightly different sizes on the two clusters.

   This is truly strange. PETSc has always used the exact same format for its binary files and automatically handles any byte-swapping that may be need to always have a consistent representation on the disk. Also a difference in the file size is strange. In over 20 years we've never had an issue with unreadable binary files.

  Can cluster 1 read the binary files from cluster 1 and cluster 2 read the binary files from cluster 2. Is it just cluster 2 cannot read the files from cluster 1?

  Could the file be changed somehow as it is copied between the clusters?

  Is this difference in size reproducible if you delete the files and create them again?

  Is one of the clusters by any chance a Microsoft Windows cluster?

Is one configured with 64-bit integers and the other not?

  Thanks,

     Matt

   Barry

> Can you comment on what I need to do to be able to read binary files on cluster 2.
>
> thanks
> Sanjay
>
> Code snippet from parallel petsc code that does output:
>
> if(time_int % (int)(1.0/DELTAT) == 0){ // a smaller time step and more files outputted makes the CV estimate better.
> sprintf(str,"my_2d%d.vts", file_Counter); // this confirms that simulation does something meaningful.
> PetscViewer viewer;  PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERVTK);
> PetscViewerFileSetName(viewer, str); VecView(u, viewer); PetscViewerDestroy(&viewer);
>                 sprintf(str,"my_3d%d.bin",(int)file_Counter);
> PetscViewer viewer2;
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE,&viewer2);
> VecView(u,viewer2);
> PetscViewerDestroy(&viewer2);
> file_Counter++;
> }
>
>
> How I am trying to read it (typically serial code with binary called ecg):
>
>   // inputs are Petsc binary files, create and destroy viewer at each file for simplicity.
> PetscViewer viewer_in;
> sprintf(str,"my_3d%d.bin",file_Counter);
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,str,FILE_MODE_READ,&viewer_in);
> VecLoad(u,viewer_in);
> PetscViewerDestroy(&viewer_in);
>
>
> Errors I got when I ran ecg:
>
> login3 endo]$ ./ecg
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Not a vector next in file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017
> [0]PETSC ERROR: ./ecg on a arch-linux2-c-opt named gra-login3 by kharches Tue Feb  5 17:43:36 2019
> [0]PETSC ERROR: Configure options --prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/petsc/3.7.5<http://soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/petsc/3.7.5> --with-mkl_pardiso=1 --with-mkl_pardiso-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl<http://soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl> --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18<http://soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18> --download-hypre=1 --download-metis=1 --download-triangle=1 --download-ptscotch=1 --download-superlu_dist=1 --download-ml=1 --download-superlu=1 --download-prometheus=1 --download-mumps=1 --download-parmetis=1 --download-suitesparse=1 --download-mumps-shared=0 --download-ptscotch-shared=0 --download-superlu-shared=0 --download-superlu_dist-shared=0 --download-parmetis-shared=0 --download-metis-shared=0 --download-ml-shared=0 --download-suitesparse-shared=0 --download-hypre-shared=0 --download-prometheus-shared=0 --with-cc=mpicc --with-cxx=mpicxx --with-c++-support --with-fc=mpifort --CFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --CXXFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --FFLAGS="-O2 -xCore-AVX2 -ftz -fp-speculation=safe -fp-model source -fPIC" --with-gnu-compilers=0 --with-mpi=1 --with-build-step-np=8 --with-shared-libraries=1 --with-debugging=0 --with-pic=1 --with-x=0 --with-windows-graphics=0 --with-scalapack=1 --with-scalapack-include=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/include<http://soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/include> --with-scalapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_scalapack_lp64.a,libmkl_blacs_openmpi_lp64.a,libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a<http://soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_scalapack_lp64.a,libmkl_blacs_openmpi_lp64.a,libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a>]" --with-blas-lapack-lib="[/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a<http://soft.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64/libmkl_intel_lp64.a,libmkl_sequential.a,libmkl_core.a>]" --with-hdf5=1 --with-hdf5-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18<http://soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/hdf5-mpi/1.8.18> --with-fftw=1 --with-fftw-dir=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/fftw-mpi/3.3.6<http://soft.computecanada.ca/easybuild/software/2017/avx2/MPI/intel2016.4/openmpi2.1/fftw-mpi/3.3.6>
> [0]PETSC ERROR: #1 PetscViewerBinaryReadVecHeader_Private() line 28 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #2 VecLoad_Binary() line 90 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #3 VecLoad_Default() line 413 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/utils/vecio.c
> [0]PETSC ERROR: #4 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #5 VecLoad_Binary_DA() line 931 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> [0]PETSC ERROR: #6 VecLoad_Default_DA() line 964 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/dm/impls/da/gr2.c
> [0]PETSC ERROR: #7 VecLoad() line 975 in /dev/shm/ebuser/PETSc/3.7.5/iomkl-2016.4.11/petsc-3.7.5/src/vec/vec/interface/vector.c
>
> This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.

From M.Tiberga at tudelft.nl  Wed Feb  6 09:56:55 2019
From: M.Tiberga at tudelft.nl (Marco Tiberga)
Date: Wed, 6 Feb 2019 15:56:55 +0000
Subject: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS and
 PCFieldSplitGetIS?
Message-ID: <c125837dacf7429abc58b962716e0949@tudelft.nl>

Dear PETSc developers,

I am developing a coupled solver for the incompressible Navier-Stokes equations, based on the PCFIELDSPLIT implemented in PETSc (I'm using version 3.10.1).
My code is written in Fortran.

I tried to follow as much as possible this example<https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html>, but I am getting weird errors. I think the problem is in the call to PCFieldSplitSetIS.
To debug the code, for each field I wanted to call PCFieldSplitGetIS (after setting it) and print the IS with ISView. However, the compiler says the reference to PCFieldSplitGetIS is undefined.

So, I was wondering whether PCFieldSplitGetIS and PCFieldSplitSetIS are actually available for Fortran.
I tried to had a look at petsc-3.10.1/src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90, but I cannot find the interfaces for these functions.

Thanks for the clarification.

Best regards,

Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/678e5c92/attachment.html>

From jychang48 at gmail.com  Wed Feb  6 14:56:12 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 Feb 2019 13:56:12 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
Message-ID: <CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>

Hi Shri,

Yeah so here at NREL I'm working with a group of folks (specifically
Dheepak Krishnamurthy, who apparently knows you back when y'all were both
at Argonne) and asked them to cook up a urban-suburban distribution system
from OpenDSS (he wrote a Julia/Python wrapper for OpenDSS). I used the
following options:

-ksp_type gmres
-pc_type asm
-pc_asm_overlap 3
-sub_pc_type lu
-sub_pc_factor_mat_ordering_type amd # using rcm also produced similar
results, all of which better than the default
-ksp_monitor_true_residual
-ksp_view
-log_view

And here are the long overdue outputs for up to 16 MPI processes on our
Eagle HPC system (with Skylake server nodes).

Basically, the algorithmic convergence degrades when more MPI procs are
used, and while I could ameliorate the KSP count by increasing the overlap,
it slowdowns the overall computation which I assume is from the
communication latencies. Also note that the KSPSolve time also slowed down
because of the ksp_monitor

Is this a "big performance hit"? I don't know, maybe I need to first start
off with a fully 3-phase system and see if similar solver characteristics
are found?

Thanks,
Justin





On Tue, Feb 5, 2019 at 6:09 PM Abhyankar, Shrirang G <
shrirang.abhyankar at pnnl.gov> wrote:

> Hi Justin,
>
>    Typically, the power grid distribution systems have a radial structure
> (unless it is an urban area) that leads to a, more or less, staircase type
> matrix. So a MatLoad() or VecLoad() would presumably just just splits the
> stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the
> phases for a bus to be split across processors, but I don?t think it would
> cause a big performance hit.
>
> I?ve worked with OpenDSS recently and have created a C wrapper for it that
> you can use to query values from it. I can help out with your
> experimentations, if you need.
>
>
>
> Shri
>
> *From:* petsc-users <petsc-users-bounces at mcs.anl.gov> * On Behalf Of *Justin
> Chang via petsc-users
> *Sent:* Tuesday, February 5, 2019 5:23 PM
> *To:* Mark Adams <mfadams at lbl.gov>
> *Cc:* petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Preconditioning systems of equations with
> complex numbers
>
>
>
> Hi all,
>
>
>
> So I *think* I know what the problem is. I am modeling a a distribution
> system where each bus can have anywhere from 1 to 3 phases. In the smallest
> example I had, all four buses had 3 phases (hence 12 dofs), and this was
> very easy to decompose - each MPI process should have a specific set of
> complex voltages on their respective nodes/buses. The larger test cases
> have lots of single phase buses attached, so a naive decomposition of the
> matrix isn't trivial. Again we're only extracting the Y-bus matrices out of
> an external software (OpenDSS to be specific) so we have no a priori
> information on how they organize their degrees-of-freedom - we let
> MatLoad() and VecLoad() do the decomposition. So it's possible that during
> this decomposition, a single bus' phases are being split across two or more
> MPI processes, which I assume would mess up the algorithmic performance of
> GAMG/ASM when more than one MPI process is needed.
>
>
>
> Yeah DMNetwork will fix this and ensure that all dofs per vertex do not
> get split across different MPI procs, but for now we're just trying to see
> if we can get a simple proof-of-concept working. We're trying to come up
> with a toy distribution system where everything is three-phase (hence very
> straightforward to decompose even with MatLoad().
>
>
>
> Justin
>
>
>
> On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov> wrote:
>
> I would stay away from eigen estimates in the solver (but give us the
> spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.
>
>
>
> Applications that have lived on direct solvers can add sorts of crap like
> penalty terms.
>
>
>
> sor seemed to work OK so I'd check the coarse grids in GAMG. Test with
> just two levels. That way you don't have to use sor on an internal MG grid,
> which can be bad and probably is here.
>
>
>
>
>
> On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>
>
> > On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Thanks everyone for your suggestions/feedback. So a few things:
> >
> > 1) When I examined larger distribution networks (~150k buses) my
> eigenvalue estimates from the chebyshev method get enormous indeed. See
> below:
> >
> > [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
> nnz/row (ave)=6, np=1
> > [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with
> threshold 0., 6.01293 nnz ave. (N=320745)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
> min=4.639523e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with
> threshold 0., 7.32435 nnz ave. (N=44797)
> > [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
> min=9.687506e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1
> active pes
> > [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with
> threshold 0., 22.5607 nnz ave. (N=13043)
> > [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
> min=6.150856e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active
> pes
> > [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with
> threshold 0., 79.5142 nnz ave. (N=2256)
> > [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
> min=6.780909e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with
> threshold 0., 206.379 nnz ave. (N=232)
> > [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
> min=2.153627e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
> >
> > 2) I tried all the suggestions mentioned before: setting
> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
> iterations of richardson/sor.
> >
> > 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just
> solving LU on the whole system.
> >
> > 3b)The problem is that I can't get it to even speedup across 2 MPI
> processes for my 150k bus case (~300k dofs) - and I already checked that
> this problem could in theory be parallelized by setting the -ksp_max_it to
> something low and observing that the KSPSolve time decreases as MPI
> concurrency increases. The potential speedup is countered by the fact that
> the algorithmic convergence rate blows up when either more MPI processes
> are added or when I tune the block_size/overlap parameters.
>
>    Yes, we just haven't gotten a scalable preconditioner yet for your
> matrix.
>
> >
> > Which leads me to one of two conclusions/questions:
> >
> > A) Is my 300 DOF problem still too small? Do I need to look at a problem
> with, say, 10 Million DOF or more to see that increasing the asm_block_size
> will have better performance (despite having more KSP iterations) than a
> single giant asm_block?
>
>    No, if you had a scalable preconditioner you would see good speed up
> with 300k unknowns, using 10 million won't help.
>
> >
> > B) Or is there something even more sinister about my system of equations
> in complex form?
>
>   There is a technical term for matrices for which both GAMG and ASM do
> very poorly on: nasty :-)
>
>   Have you tried using parallel LU (for example ./configure
> --download-superlu_dist). Direct LU is the last refuge of the
> unpreconditionable. How large will your matrices get?
>
>    Barry
>
> >
> > Thanks,
> > Justin
> >
> > On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > The coarse grid is getting set to SOR rather than LU.
> >
> >    Matt
> >
> > On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
> > I tried these options:
> >
> > -ksp_view
> > -ksp_monitor_true_residual
> > -ksp_type gmres
> > -ksp_max_it 50
> > -pc_type gamg
> > -mg_coarse_pc_type sor
> > -mg_levels_1_ksp_type richardson
> > -mg_levels_2_ksp_type richardson
> > -mg_levels_3_ksp_type richardson
> > -mg_levels_4_ksp_type richardson
> > -mg_levels_1_pc_type sor
> > -mg_levels_2_pc_type sor
> > -mg_levels_3_pc_type sor
> > -mg_levels_4_pc_type sor
> >
> > And still have a non-converging solution:
> >
> >   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
> >   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
> >   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
> >   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
> >   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
> >   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
> >   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
> >   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
> >   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> >  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> >  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> >  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> >  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> >  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> >  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> >  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> >  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> >  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> >  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> >  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> >  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> >  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> >  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> >  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> >  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> >  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> >  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> >  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> >  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> >  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> >  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> >  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> >  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> >  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> >  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> >  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> >  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> >  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> >  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> >  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> >  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> >  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> >  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> >  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> >  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> >  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> >  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> >  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> >  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> >  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> > KSP Object: 1 MPI processes
> >   type: gmres
> >     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >     happy breakdown tolerance 1e-30
> >   maximum iterations=50, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: gamg
> >     type is MULTIPLICATIVE, levels=5 cycles=v
> >       Cycles per PCApply=1
> >       Using externally compute Galerkin coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values in graph on each level =
>  0.   0.   0.
> >         Threshold scaling factor for each level not specified = 1.
> >         AGG specific options
> >           Symmetric graph false
> >           Number of levels to square graph 1
> >           Number smoothing steps 1
> >         Complexity:    grid = 1.31821
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object: (mg_coarse_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_coarse_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=17, cols=17
> >         total: nonzeros=163, allocated nonzeros=163
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 12 nodes, limit used is 5
> >   Down solver (pre-smoother) on level 1 -------------------------------
> >     KSP Object: (mg_levels_1_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_1_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=100, cols=100
> >         total: nonzeros=1240, allocated nonzeros=1240
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 2 -------------------------------
> >     KSP Object: (mg_levels_2_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_2_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=537, cols=537
> >         total: nonzeros=5291, allocated nonzeros=5291
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 3 -------------------------------
> >     KSP Object: (mg_levels_3_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_3_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=1541, cols=1541
> >         total: nonzeros=8039, allocated nonzeros=8039
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 4 -------------------------------
> >     KSP Object: (mg_levels_4_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_4_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=8541, cols=8541
> >         total: nonzeros=46299, allocated nonzeros=46299
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 5464 nodes, limit used is 5
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> >     type: seqaij
> >     rows=8541, cols=8541
> >     total: nonzeros=46299, allocated nonzeros=46299
> >     total number of mallocs used during MatSetValues calls =0
> >       using I-node routines: found 5464 nodes, limit used is 5
> >
> > Am I doing this right? Did I miss anything?
> >
> >
> >
> > On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
> > Hi Mark,
> >
> > 1) So with these options:
> >
> > -ksp_type gmres
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -ksp_converged_reason
> > -pc_type bjacobi
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
> >   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
> >   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
> >   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
> >   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 5
> >
> > 2) With richardson/sor:
> >
> > Okay, its looks like Richardson/SOR solves this just fine. You can use
> this as the smoother for GAMG instead
> > of Cheby/Jacobi, and probably see better results on the larger problems.
> >
> >   Matt
> >
> > -ksp_type richardson
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type sor
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
> >   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
> >   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
> >   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
> >   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
> >   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
> >   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
> >   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
> >   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
> >  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
> >  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
> >  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
> >  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
> >  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
> >  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
> >  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
> >  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
> >  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
> >  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
> >  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
> >  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
> >  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
> >  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
> >  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
> >  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
> >  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
> >  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
> >  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
> >  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
> >  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
> >  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
> >  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
> >  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
> >  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
> >  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
> >  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
> >  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
> >  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
> >  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
> >  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 40
> >
> > 3) And lastly with chebyshev/jacobi:
> >
> > -ksp_type chebyshev
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type jacobi
> >
> >   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
> >   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
> >   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
> >   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
> >   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
> >   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
> >   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
> >   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
> >   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
> >   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
> >  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
> >  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
> >  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
> >  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
> >  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
> >  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
> >  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
> >  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
> >  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
> >  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
> >  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
> >  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
> >  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
> >  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
> >  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
> >  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
> >  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
> >  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
> >  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
> >  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
> >  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
> >  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
> >  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
> >  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
> >  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
> >  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
> >  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
> >  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
> >  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
> >  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
> >  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
> >  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
> >  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
> >  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
> >  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
> >  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
> >  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
> >  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
> >  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
> >  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
> >  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
> >  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
> >  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
> >  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
> >  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
> >  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
> >  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
> >  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
> >  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
> >  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
> >  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
> >  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
> >  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
> >  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
> >  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
> >  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
> >  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 67
> >
> > Looks like neither of these ksp/pc combos are good? I also tried
> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
> GAMG info I was able to grep from the out-of-box params:
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
> min=2.394056e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 5.21674 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
> min=6.783380e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 9.85289 nnz ave. (N=537)
> > [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
> min=5.974776e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 12.4 nnz ave. (N=100)
> > [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
> min=4.842076e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> >
> > And the one with pc_gamg_agg_nsmooths 0
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with
> threshold 0., 3.07398 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> > [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.02211 nnz ave. (N=814)
> > [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> > [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.00434 nnz ave. (N=461)
> > [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> > [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=290)
> > [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> > [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=197)
> > [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> > [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.98425 nnz ave. (N=127)
> > [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> > [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.97561 nnz ave. (N=82)
> > [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> > [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.9697 nnz ave. (N=66)
> > [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> > [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> >
> >
> > Thanks,
> > Justin
> >
> > On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> >
> >
> > Both GAMG and ILU are nice and dandy for this,
> >
> > I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
> converge it way down, say rtol = 1.e-12. See which one is better in the
> early iteration and pick it. It would be nice to check that it solves the
> problem ...
> >
> > The residual drops about 5 orders in the first iteration and then
> flatlines. That is very bad. Check that the smoother can actually solve the
> problem.
> >
> > but as soon as I look at a bigger system, like a network with 8500
> buses, the out-of-box gamg craps out. I am not sure where to start when it
> comes to tuning GAMG.
> >
> > First try -pc_gamg_nsmooths 0
> >
> > You can run with -info and grep on GAMG to see some diagnostic output.
> Eigen estimates are fragile in practice and with this parameter and SOR
> there are no eigen estimates needed. The max eigen values for all levels
> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
> problem.
> >
> > Attached is the ksp monitor/view output for gamg on the unsuccessful
> solve
> >
> > I'm also attaching a zip file which contains the simple PETSc script
> that loads the binary matrix/vector as well as two test cases, if you guys
> are interested in trying it out. It only works if you have PETSc configured
> with complex numbers.
> >
> > Thanks
> >
> > Justin
> >
> > PS - A couple years ago I had asked if there was a paper/tutorial on
> using/tuning GAMG. Does such a thing exist today?
> >
> > There is a write up in the manual that is tutorial like.
> >
> >
> > On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
> wrote:
> > Here's IMHO the simplest explanation of the equations I'm trying to
> solve:
> >
> > http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> >
> > Right now we're just trying to solve eq(5) (in section 1), inverting the
> linear Y-bus matrix. Eventually we have to be able to solve equations like
> those in the next section.
> >
> > Maybe I am reading this wrong, but the Y-bus matrix looks like an
> M-matrix to me (if all the y's are positive). This means
> > that it should be really easy to solve, and I think GAMG should do it.
> You can start out just doing relaxation, like SOR, on
> > small examples.
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi all,
> >
> > I'm working with some folks to extract a linear system of equations from
> an external software package that solves power flow equations in complex
> form. Since that external package uses serial direct solvers like KLU from
> suitesparse, I want a proof-of-concept where the same matrix can be solved
> in PETSc using its parallel solvers.
> >
> > I got mumps to achieve a very minor speedup across two MPI processes on
> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
> seconds). However I want to use iterative solvers and preconditioners but I
> have never worked with complex numbers so I am not sure what the "best"
> options are given PETSc's capabilities.
> >
> > So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
> believe I also tried BICG with BJACOBI and while it did converge it
> converged slowly. Does anyone have recommendations on how one would go
> about preconditioning PETSc matrices with complex numbers? I was originally
> thinking about converting it to cartesian form: Declaring all voltages =
> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
> arctan(imaginary/real) because all the papers I've seen in literature that
> claim to successfully precondition power flow equations operate in this
> form.
> >
> > 1) We really need to see the (simplified) equations
> >
> > 2) All complex equations can be converted to a system of real equations
> twice as large, but this is not necessarily the best way to go
> >
> >  Thanks,
> >
> >     Matt
> >
> > Justin
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: case1MPI
Type: application/octet-stream
Size: 18774 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: case4MPI
Type: application/octet-stream
Size: 21512 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: case8MPI
Type: application/octet-stream
Size: 29317 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: case16MPI
Type: application/octet-stream
Size: 37732 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: case2MPI
Type: application/octet-stream
Size: 20776 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190206/80f70d6a/attachment-0009.obj>

From bsmith at mcs.anl.gov  Wed Feb  6 22:18:52 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 7 Feb 2019 04:18:52 +0000
Subject: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS
 and PCFieldSplitGetIS?
In-Reply-To: <c125837dacf7429abc58b962716e0949@tudelft.nl>
References: <c125837dacf7429abc58b962716e0949@tudelft.nl>
Message-ID: <02C8FA4C-F42A-40A5-ADCD-2D253FACB426@anl.gov>


   Marco,
   We were missing a Fortran stub in the code. You can either apply the attached patch or use the git branch barry/add-pcfieldsplit-getis/maint for the missing functionality. The fix will also be in the next maintenance release of petsc-3.10

   Please let us know if there are any difficulties.

   Barry


> On Feb 6, 2019, at 9:56 AM, Marco Tiberga via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Dear PETSc developers,
>
> I am developing a coupled solver for the incompressible Navier-Stokes equations, based on the PCFIELDSPLIT implemented in PETSc (I?m using version 3.10.1).
> My code is written in Fortran.
>
> I tried to follow as much as possible this example, but I am getting weird errors. I think the problem is in the call to PCFieldSplitSetIS.
> To debug the code, for each field I wanted to call PCFieldSplitGetIS (after setting it) and print the IS with ISView. However, the compiler says the reference to PCFieldSplitGetIS is undefined.
>
> So, I was wondering whether PCFieldSplitGetIS and PCFieldSplitSetIS are actually available for Fortran.
> I tried to had a look at petsc-3.10.1/src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90, but I cannot find the interfaces for these functions.
>
> Thanks for the clarification.
>
> Best regards,
>
> Marco Tiberga
> PhD candidate
> Delft University of Technology
> Faculty of Applied Sciences
> Radiation Science & Technology Department
> Mekelweg 15, 2629 JB Delft, The Netherlands
> E-Mail: m.tiberga at tudelft.nl
> Website: http://www.nera.rst.tudelft.nl/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190207/88abf3c8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: add-pcfieldsplit-getis.patch
Type: application/octet-stream
Size: 2993 bytes
Desc: add-pcfieldsplit-getis.patch
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190207/88abf3c8/attachment.obj>

From M.Tiberga at tudelft.nl  Thu Feb  7 08:31:28 2019
From: M.Tiberga at tudelft.nl (Marco Tiberga)
Date: Thu, 7 Feb 2019 14:31:28 +0000
Subject: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS
 and PCFieldSplitGetIS?
In-Reply-To: <02C8FA4C-F42A-40A5-ADCD-2D253FACB426@anl.gov>
References: <c125837dacf7429abc58b962716e0949@tudelft.nl>
	<02C8FA4C-F42A-40A5-ADCD-2D253FACB426@anl.gov>
Message-ID: <117f6124891545ef9695bc952a12a67d@tudelft.nl>

Dear Barry,

Thanks a lot for your fast reply, as usual.
The patch did work!

With kind regards,
Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/



From: Smith, Barry F. [mailto:bsmith at mcs.anl.gov]
Sent: donderdag 7 februari 2019 5:19
To: Marco Tiberga
Cc: petsc-users at mcs.anl.gov; Danny Lathouwers - TNW
Subject: Re: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS and PCFieldSplitGetIS?


   Marco,
   We were missing a Fortran stub in the code. You can either apply the attached patch or use the git branch barry/add-pcfieldsplit-getis/maint for the missing functionality. The fix will also be in the next maintenance release of petsc-3.10

   Please let us know if there are any difficulties.

   Barry


> On Feb 6, 2019, at 9:56 AM, Marco Tiberga via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Dear PETSc developers,
>
> I am developing a coupled solver for the incompressible Navier-Stokes equations, based on the PCFIELDSPLIT implemented in PETSc (I?m using version 3.10.1).
> My code is written in Fortran.
>
> I tried to follow as much as possible this example, but I am getting weird errors. I think the problem is in the call to PCFieldSplitSetIS.
> To debug the code, for each field I wanted to call PCFieldSplitGetIS (after setting it) and print the IS with ISView. However, the compiler says the reference to PCFieldSplitGetIS is undefined.
>
> So, I was wondering whether PCFieldSplitGetIS and PCFieldSplitSetIS are actually available for Fortran.
> I tried to had a look at petsc-3.10.1/src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90, but I cannot find the interfaces for these functions.
>
> Thanks for the clarification.
>
> Best regards,
>
> Marco Tiberga
> PhD candidate
> Delft University of Technology
> Faculty of Applied Sciences
> Radiation Science & Technology Department
> Mekelweg 15, 2629 JB Delft, The Netherlands
> E-Mail: m.tiberga at tudelft.nl
> Website: http://www.nera.rst.tudelft.nl/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190207/1b77ce8c/attachment.html>

From shrirang.abhyankar at pnnl.gov  Thu Feb  7 09:02:54 2019
From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G)
Date: Thu, 7 Feb 2019 15:02:54 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
Message-ID: <DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>

Hi Justin,
    The serial solve time for this matrix of size 320K is about 0.3 secs, which I think is pretty fast. As such, I am not surprised you are not getting any speed up in parallel with asm. Perhaps, you need a bigger matrix?

It would be interesting to see a spy plot of the matrix on different processors. If there are a lot of off-diagonal elements, then asm may not work that well. You may need to use a partitioner in that case.

As a side note, folks in the circuits community consider KLU as one of the fastest serial solvers for such kind of problems. You can try that out and see if it improves serial performance even further.

--download-suitesparse

-pc_factor_mat_solver_package klu -pc_factor_mat_ordering_type amd

Shri

From: Justin Chang <jychang48 at gmail.com>
Sent: Wednesday, February 6, 2019 2:56 PM
To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
Cc: Mark Adams <mfadams at lbl.gov>; PETSc users list <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

Hi Shri,

Yeah so here at NREL I'm working with a group of folks (specifically Dheepak Krishnamurthy, who apparently knows you back when y'all were both at Argonne) and asked them to cook up a urban-suburban distribution system from OpenDSS (he wrote a Julia/Python wrapper for OpenDSS). I used the following options:

-ksp_type gmres
-pc_type asm
-pc_asm_overlap 3
-sub_pc_type lu
-sub_pc_factor_mat_ordering_type amd # using rcm also produced similar results, all of which better than the default
-ksp_monitor_true_residual
-ksp_view
-log_view

And here are the long overdue outputs for up to 16 MPI processes on our Eagle HPC system (with Skylake server nodes).

Basically, the algorithmic convergence degrades when more MPI procs are used, and while I could ameliorate the KSP count by increasing the overlap, it slowdowns the overall computation which I assume is from the communication latencies. Also note that the KSPSolve time also slowed down because of the ksp_monitor

Is this a "big performance hit"? I don't know, maybe I need to first start off with a fully 3-phase system and see if similar solver characteristics are found?

Thanks,
Justin





On Tue, Feb 5, 2019 at 6:09 PM Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov<mailto:shrirang.abhyankar at pnnl.gov>> wrote:
Hi Justin,
   Typically, the power grid distribution systems have a radial structure (unless it is an urban area) that leads to a, more or less, staircase type matrix. So a MatLoad() or VecLoad() would presumably just just splits the stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the phases for a bus to be split across processors, but I don?t think it would cause a big performance hit.
I?ve worked with OpenDSS recently and have created a C wrapper for it that you can use to query values from it. I can help out with your experimentations, if you need.

Shri
From: petsc-users <petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>> On Behalf Of Justin Chang via petsc-users
Sent: Tuesday, February 5, 2019 5:23 PM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

Hi all,

So I *think* I know what the problem is. I am modeling a a distribution system where each bus can have anywhere from 1 to 3 phases. In the smallest example I had, all four buses had 3 phases (hence 12 dofs), and this was very easy to decompose - each MPI process should have a specific set of complex voltages on their respective nodes/buses. The larger test cases have lots of single phase buses attached, so a naive decomposition of the matrix isn't trivial. Again we're only extracting the Y-bus matrices out of an external software (OpenDSS to be specific) so we have no a priori information on how they organize their degrees-of-freedom - we let MatLoad() and VecLoad() do the decomposition. So it's possible that during this decomposition, a single bus' phases are being split across two or more MPI processes, which I assume would mess up the algorithmic performance of GAMG/ASM when more than one MPI process is needed.

Yeah DMNetwork will fix this and ensure that all dofs per vertex do not get split across different MPI procs, but for now we're just trying to see if we can get a simple proof-of-concept working. We're trying to come up with a toy distribution system where everything is three-phase (hence very straightforward to decompose even with MatLoad().

Justin

On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
I would stay away from eigen estimates in the solver (but give us the spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.

Applications that have lived on direct solvers can add sorts of crap like penalty terms.

sor seemed to work OK so I'd check the coarse grids in GAMG. Test with just two levels. That way you don't have to use sor on an internal MG grid, which can be bad and probably is here.


On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


> On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Thanks everyone for your suggestions/feedback. So a few things:
>
> 1) When I examined larger distribution networks (~150k buses) my eigenvalue estimates from the chebyshev method get enormous indeed. See below:
>
> [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1, nnz/row (ave)=6, np=1
> [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with threshold 0., 6.01293 nnz ave. (N=320745)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00 min=4.639523e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
> [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with threshold 0., 7.32435 nnz ave. (N=44797)
> [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00 min=9.687506e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active pes
> [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with threshold 0., 22.5607 nnz ave. (N=13043)
> [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00 min=6.150856e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
> [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with threshold 0., 79.5142 nnz ave. (N=2256)
> [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00 min=6.780909e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
> [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with threshold 0., 206.379 nnz ave. (N=232)
> [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00 min=2.153627e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
>
> 2) I tried all the suggestions mentioned before: setting -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more iterations of richardson/sor.
>
> 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just solving LU on the whole system.
>
> 3b)The problem is that I can't get it to even speedup across 2 MPI processes for my 150k bus case (~300k dofs) - and I already checked that this problem could in theory be parallelized by setting the -ksp_max_it to something low and observing that the KSPSolve time decreases as MPI concurrency increases. The potential speedup is countered by the fact that the algorithmic convergence rate blows up when either more MPI processes are added or when I tune the block_size/overlap parameters.

   Yes, we just haven't gotten a scalable preconditioner yet for your matrix.

>
> Which leads me to one of two conclusions/questions:
>
> A) Is my 300 DOF problem still too small? Do I need to look at a problem with, say, 10 Million DOF or more to see that increasing the asm_block_size will have better performance (despite having more KSP iterations) than a single giant asm_block?

   No, if you had a scalable preconditioner you would see good speed up with 300k unknowns, using 10 million won't help.

>
> B) Or is there something even more sinister about my system of equations in complex form?

  There is a technical term for matrices for which both GAMG and ASM do very poorly on: nasty :-)

  Have you tried using parallel LU (for example ./configure --download-superlu_dist). Direct LU is the last refuge of the unpreconditionable. How large will your matrices get?

   Barry

>
> Thanks,
> Justin
>
> On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> The coarse grid is getting set to SOR rather than LU.
>
>    Matt
>
> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> I tried these options:
>
> -ksp_view
> -ksp_monitor_true_residual
> -ksp_type gmres
> -ksp_max_it 50
> -pc_type gamg
> -mg_coarse_pc_type sor
> -mg_levels_1_ksp_type richardson
> -mg_levels_2_ksp_type richardson
> -mg_levels_3_ksp_type richardson
> -mg_levels_4_ksp_type richardson
> -mg_levels_1_pc_type sor
> -mg_levels_2_pc_type sor
> -mg_levels_3_pc_type sor
> -mg_levels_4_pc_type sor
>
> And still have a non-converging solution:
>
>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=5 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.   0.   0.
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>         Complexity:    grid = 1.31821
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=17, cols=17
>         total: nonzeros=163, allocated nonzeros=163
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 12 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=100, cols=100
>         total: nonzeros=1240, allocated nonzeros=1240
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=537, cols=537
>         total: nonzeros=5291, allocated nonzeros=5291
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=1541, cols=1541
>         total: nonzeros=8039, allocated nonzeros=8039
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=8541, cols=8541
>         total: nonzeros=46299, allocated nonzeros=46299
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 5464 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=8541, cols=8541
>     total: nonzeros=46299, allocated nonzeros=46299
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 5464 nodes, limit used is 5
>
> Am I doing this right? Did I miss anything?
>
>
>
> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Hi Mark,
>
> 1) So with these options:
>
> -ksp_type gmres
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -ksp_converged_reason
> -pc_type bjacobi
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> Linear solve converged due to CONVERGED_RTOL iterations 5
>
> 2) With richardson/sor:
>
> Okay, its looks like Richardson/SOR solves this just fine. You can use this as the smoother for GAMG instead
> of Cheby/Jacobi, and probably see better results on the larger problems.
>
>   Matt
>
> -ksp_type richardson
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type sor
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> Linear solve converged due to CONVERGED_RTOL iterations 40
>
> 3) And lastly with chebyshev/jacobi:
>
> -ksp_type chebyshev
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type jacobi
>
>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> Linear solve converged due to CONVERGED_RTOL iterations 67
>
> Looks like neither of these ksp/pc combos are good? I also tried -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the GAMG info I was able to grep from the out-of-box params:
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00 min=2.394056e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 5.21674 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00 min=6.783380e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 9.85289 nnz ave. (N=537)
> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00 min=5.974776e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 12.4 nnz ave. (N=100)
> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00 min=4.842076e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>
> And the one with pc_gamg_agg_nsmooths 0
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with threshold 0., 3.07398 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.02211 nnz ave. (N=814)
> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.00434 nnz ave. (N=461)
> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=290)
> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=197)
> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.98425 nnz ave. (N=127)
> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.97561 nnz ave. (N=82)
> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.9697 nnz ave. (N=66)
> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>
>
> Thanks,
> Justin
>
> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
>
>
> Both GAMG and ILU are nice and dandy for this,
>
> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and converge it way down, say rtol = 1.e-12. See which one is better in the early iteration and pick it. It would be nice to check that it solves the problem ...
>
> The residual drops about 5 orders in the first iteration and then flatlines. That is very bad. Check that the smoother can actually solve the problem.
>
> but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG.
>
> First try -pc_gamg_nsmooths 0
>
> You can run with -info and grep on GAMG to see some diagnostic output. Eigen estimates are fragile in practice and with this parameter and SOR there are no eigen estimates needed. The max eigen values for all levels should be between say 2 (or less) and say 3-4. Much higher is a sign of a problem.
>
> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
>
> I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
>
> Thanks
>
> Justin
>
> PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
>
> There is a write up in the manual that is tutorial like.
>
>
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>
> Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
>
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> small examples.
>
>   Thanks,
>
>     Matt
>
> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> Hi all,
>
> I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers.
>
> I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
>
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
>
> 1) We really need to see the (simplified) equations
>
> 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
>
>  Thanks,
>
>     Matt
>
> Justin
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190207/e81cbb9b/attachment-0001.html>

From danyang.su at gmail.com  Thu Feb  7 15:04:03 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Thu, 7 Feb 2019 13:04:03 -0800
Subject: [petsc-users] different dof in DMDA creating
Message-ID: <d54af890-839a-af1c-238f-983ade4b3020@gmail.com>

Dear PETSc Users,

Does DMDA support different number of degrees of freedom for different 
node? For example I have a 2D subsurface flow problem with the default 
dof = 1 throughout the domain. Now I want to add some sparse fractures 
in the domain. For the nodes connected to the sparse fractures, I want 
to set dof to 2. Is it possible to set dof to 2 for those nodes only?

Thanks,

Danyang


From bsmith at mcs.anl.gov  Thu Feb  7 15:14:06 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 7 Feb 2019 21:14:06 +0000
Subject: [petsc-users] different dof in DMDA creating
In-Reply-To: <d54af890-839a-af1c-238f-983ade4b3020@gmail.com>
References: <d54af890-839a-af1c-238f-983ade4b3020@gmail.com>
Message-ID: <F0C4257F-1270-4421-9F88-42E86C03C281@anl.gov>


  No, you would need to use the more flexible DMPlex


> On Feb 7, 2019, at 3:04 PM, Danyang Su via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear PETSc Users,
> 
> Does DMDA support different number of degrees of freedom for different node? For example I have a 2D subsurface flow problem with the default dof = 1 throughout the domain. Now I want to add some sparse fractures in the domain. For the nodes connected to the sparse fractures, I want to set dof to 2. Is it possible to set dof to 2 for those nodes only?
> 
> Thanks,
> 
> Danyang
> 


From danyang.su at gmail.com  Thu Feb  7 15:53:06 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Thu, 7 Feb 2019 13:53:06 -0800
Subject: [petsc-users] different dof in DMDA creating
In-Reply-To: <F0C4257F-1270-4421-9F88-42E86C03C281@anl.gov>
References: <d54af890-839a-af1c-238f-983ade4b3020@gmail.com>
	<F0C4257F-1270-4421-9F88-42E86C03C281@anl.gov>
Message-ID: <6ee19350-d059-703b-9c52-b03ae6bf7a4e@gmail.com>

Thanks, Barry. DMPlex also works for my code.

Danyang

On 2019-02-07 1:14 p.m., Smith, Barry F. wrote:
>    No, you would need to use the more flexible DMPlex
>
>
>> On Feb 7, 2019, at 3:04 PM, Danyang Su via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Dear PETSc Users,
>>
>> Does DMDA support different number of degrees of freedom for different node? For example I have a 2D subsurface flow problem with the default dof = 1 throughout the domain. Now I want to add some sparse fractures in the domain. For the nodes connected to the sparse fractures, I want to set dof to 2. Is it possible to set dof to 2 for those nodes only?
>>
>> Thanks,
>>
>> Danyang
>>

From sajidsyed2021 at u.northwestern.edu  Thu Feb  7 17:40:36 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 7 Feb 2019 17:40:36 -0600
Subject: [petsc-users] Link to the latest tutorials broken
Message-ID: <CAOGsD9iuy8E=DffkiQmfGHfh2p_J+ioy3JjuMxJzWKWr6Rhd-A@mail.gmail.com>

Hi,

The links to the Jan 2019 presentations at
https://www.mcs.anl.gov/petsc/documentation/tutorials/index.html are
broken. Could these be fixed ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190207/e66e93ac/attachment.html>

From balay at mcs.anl.gov  Thu Feb  7 17:53:11 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 7 Feb 2019 23:53:11 +0000
Subject: [petsc-users] Link to the latest tutorials broken
In-Reply-To: <CAOGsD9iuy8E=DffkiQmfGHfh2p_J+ioy3JjuMxJzWKWr6Rhd-A@mail.gmail.com>
References: <CAOGsD9iuy8E=DffkiQmfGHfh2p_J+ioy3JjuMxJzWKWr6Rhd-A@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1902071753030.2295@sb>

fixed now.

Satish

On Thu, 7 Feb 2019, Sajid Ali via petsc-users wrote:

> Hi,
> 
> The links to the Jan 2019 presentations at
> https://www.mcs.anl.gov/petsc/documentation/tutorials/index.html are
> broken. Could these be fixed ?
> 
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University
> 


From eda.oktay at metu.edu.tr  Fri Feb  8 07:24:12 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Fri, 8 Feb 2019 16:24:12 +0300
Subject: [petsc-users] Compilation of program including slepc
Message-ID: <CA+vu_Xj4WOQ1Fv-QAK3nVOBkuj=FDYsEBXhi2sKHZGgQFy1Aig@mail.gmail.com>

Hello,

I installed SLEPc during configuration of PETSc and now I am trying to
write a program in PETSc including SLEPc library. For start, I just
duplicated ex11 in EPS folder in
petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eps/examples and
renamed as eda.c and tried to compile it by "make eda" and I got the
following error:



/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/bin/mpicc     eda.c   -o eda
In file included from
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcst.h:16:0,
                 from
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepceps.h:16,
                 from eda.c:17:
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcsys.h:34:22:
fatal error: petscmat.h: No such file or directory
 #include <petscmat.h>
                      ^
compilation terminated.
<builtin>: recipe for target 'eda' failed
make: *** [eda] Error 1



I didn't understand the error since ex11 was compiled successfully but eda
wasn't. Can you help me?

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/8bfc41e7/attachment.html>

From jroman at dsic.upv.es  Fri Feb  8 07:26:41 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 8 Feb 2019 14:26:41 +0100
Subject: [petsc-users] Compilation of program including slepc
In-Reply-To: <CA+vu_Xj4WOQ1Fv-QAK3nVOBkuj=FDYsEBXhi2sKHZGgQFy1Aig@mail.gmail.com>
References: <CA+vu_Xj4WOQ1Fv-QAK3nVOBkuj=FDYsEBXhi2sKHZGgQFy1Aig@mail.gmail.com>
Message-ID: <6A19F1F1-5769-4775-BF0C-2145F44C7DF9@dsic.upv.es>

You need to create a makefile for your program or add rules to an existing makefile. See section 1.4.2 of SLEPc's user manual.
Jose


> El 8 feb 2019, a las 14:24, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hello,
> 
> I installed SLEPc during configuration of PETSc and now I am trying to write a program in PETSc including SLEPc library. For start, I just duplicated ex11 in EPS folder in petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eps/examples and renamed as eda.c and tried to compile it by "make eda" and I got the following error:
> 
> 
> 
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/bin/mpicc     eda.c   -o eda
> In file included from /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcst.h:16:0,
>                  from /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepceps.h:16,
>                  from eda.c:17:
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcsys.h:34:22: fatal error: petscmat.h: No such file or directory
>  #include <petscmat.h>
>                       ^
> compilation terminated.
> <builtin>: recipe for target 'eda' failed
> make: *** [eda] Error 1
> 
> 
> 
> I didn't understand the error since ex11 was compiled successfully but eda wasn't. Can you help me?
> 
> Best regards,
> 
> Eda


From eda.oktay at metu.edu.tr  Fri Feb  8 07:42:56 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Fri, 8 Feb 2019 16:42:56 +0300
Subject: [petsc-users] Compilation of program including slepc
In-Reply-To: <6A19F1F1-5769-4775-BF0C-2145F44C7DF9@dsic.upv.es>
References: <CA+vu_Xj4WOQ1Fv-QAK3nVOBkuj=FDYsEBXhi2sKHZGgQFy1Aig@mail.gmail.com>
	<6A19F1F1-5769-4775-BF0C-2145F44C7DF9@dsic.upv.es>
Message-ID: <CA+vu_XgBQCewE9kuPNGQr-eN3y=L6Dt2OHhoKULR3XgCP3Mw2w@mail.gmail.com>

Dear Professor Roman,

Thank you for your answer.

Jose E. Roman <jroman at dsic.upv.es>, 8 ?ub 2019 Cum, 16:26 tarihinde ?unu
yazd?:

> You need to create a makefile for your program or add rules to an existing
> makefile. See section 1.4.2 of SLEPc's user manual.
> Jose
>
>
> > El 8 feb 2019, a las 14:24, Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> escribi?:
> >
> > Hello,
> >
> > I installed SLEPc during configuration of PETSc and now I am trying to
> write a program in PETSc including SLEPc library. For start, I just
> duplicated ex11 in EPS folder in
> petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eps/examples and
> renamed as eda.c and tried to compile it by "make eda" and I got the
> following error:
> >
> >
> >
> > /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/bin/mpicc     eda.c   -o
> eda
> > In file included from
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcst.h:16:0,
> >                  from
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepceps.h:16,
> >                  from eda.c:17:
> >
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/include/slepcsys.h:34:22:
> fatal error: petscmat.h: No such file or directory
> >  #include <petscmat.h>
> >                       ^
> > compilation terminated.
> > <builtin>: recipe for target 'eda' failed
> > make: *** [eda] Error 1
> >
> >
> >
> > I didn't understand the error since ex11 was compiled successfully but
> eda wasn't. Can you help me?
> >
> > Best regards,
> >
> > Eda
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/f6f5d276/attachment.html>

From eda.oktay at metu.edu.tr  Fri Feb  8 08:02:15 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Fri, 8 Feb 2019 17:02:15 +0300
Subject: [petsc-users] Problem Using EPSGetEigenpair
Message-ID: <CA+vu_Xh10CktiremVompx-C3QVF52+PQPihn39FGAc-tV1CWeA@mail.gmail.com>

Hello,

I am trying to test EPSGetEigenpair function on ex11 in EPS.

First, I executed ex11 and get 0.097887 as eigenvalue (k).

Then under the code

ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues:
%D\n",nev);CHKERRQ(ierr);

I added the following code:

  ierr = EPSGetEigenpair(eps,0,&kr,NULL,NULL,NULL);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Requested eigenvalue:
%D\n",kr);CHKERRQ(ierr);

After execution, I expected to find 0.097887 but the requested eigenvalue
was found as -876957631.

Why EPSGetEigenpair couldn't give 0.097887, isn't  0.097887 the eigenvalue
that EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL) calculated?

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/b61b3791/attachment.html>

From jroman at dsic.upv.es  Fri Feb  8 08:05:14 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 8 Feb 2019 15:05:14 +0100
Subject: [petsc-users] Problem Using EPSGetEigenpair
In-Reply-To: <CA+vu_Xh10CktiremVompx-C3QVF52+PQPihn39FGAc-tV1CWeA@mail.gmail.com>
References: <CA+vu_Xh10CktiremVompx-C3QVF52+PQPihn39FGAc-tV1CWeA@mail.gmail.com>
Message-ID: <ECA8E6A0-4943-4FDB-A1F0-EEAA1519EB9A@dsic.upv.es>

You are printing the variable as an integer, and it is not an integer, it's a PetscScalar. Use %g
Jose


> El 8 feb 2019, a las 15:02, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hello,
> 
> I am trying to test EPSGetEigenpair function on ex11 in EPS. 
> 
> First, I executed ex11 and get 0.097887 as eigenvalue (k).
> 
> Then under the code 
> 
> ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
> 
> I added the following code:
> 
>   ierr = EPSGetEigenpair(eps,0,&kr,NULL,NULL,NULL);  
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Requested eigenvalue: %D\n",kr);CHKERRQ(ierr);
> 
> After execution, I expected to find 0.097887 but the requested eigenvalue was found as -876957631.
> 
> Why EPSGetEigenpair couldn't give 0.097887, isn't  0.097887 the eigenvalue that EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL) calculated?
> 
> Best regards,
> 
> Eda


From eda.oktay at metu.edu.tr  Fri Feb  8 08:07:11 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Fri, 8 Feb 2019 17:07:11 +0300
Subject: [petsc-users] Problem Using EPSGetEigenpair
In-Reply-To: <ECA8E6A0-4943-4FDB-A1F0-EEAA1519EB9A@dsic.upv.es>
References: <CA+vu_Xh10CktiremVompx-C3QVF52+PQPihn39FGAc-tV1CWeA@mail.gmail.com>
	<ECA8E6A0-4943-4FDB-A1F0-EEAA1519EB9A@dsic.upv.es>
Message-ID: <CA+vu_Xh2domy2tFPwvoCfqeEVbnunEbS+ON1+AUqjmoRP4Az3g@mail.gmail.com>

Dear Professor Roman,

Thank you for answering.

Eda

Jose E. Roman <jroman at dsic.upv.es>, 8 ?ub 2019 Cum, 17:05 tarihinde ?unu
yazd?:

> You are printing the variable as an integer, and it is not an integer,
> it's a PetscScalar. Use %g
> Jose
>
>
> > El 8 feb 2019, a las 15:02, Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> escribi?:
> >
> > Hello,
> >
> > I am trying to test EPSGetEigenpair function on ex11 in EPS.
> >
> > First, I executed ex11 and get 0.097887 as eigenvalue (k).
> >
> > Then under the code
> >
> > ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues:
> %D\n",nev);CHKERRQ(ierr);
> >
> > I added the following code:
> >
> >   ierr = EPSGetEigenpair(eps,0,&kr,NULL,NULL,NULL);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Requested eigenvalue:
> %D\n",kr);CHKERRQ(ierr);
> >
> > After execution, I expected to find 0.097887 but the requested
> eigenvalue was found as -876957631.
> >
> > Why EPSGetEigenpair couldn't give 0.097887, isn't  0.097887 the
> eigenvalue that EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL) calculated?
> >
> > Best regards,
> >
> > Eda
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/052e06ee/attachment.html>

From adityakr42 at gmail.com  Fri Feb  8 11:40:24 2019
From: adityakr42 at gmail.com (aditya kumar)
Date: Fri, 8 Feb 2019 11:40:24 -0600
Subject: [petsc-users] Meaning of PETSc error code 77
Message-ID: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>

Hello,

I am using PETSc with FEniCS project libraries to solve a nonlinear
problem. I am using PETSc Krylov solver with the following configuration

pc = PETScPreconditioner("petsc_amg")
PETScOptions.set("mg_levels_ksp_type", "chebyshev")
PETScOptions.set("mg_levels_pc_type", "jacobi")
PETScOptions.set("mg_levels_esteig_ksp_type", "cg")
PETScOptions.set("mg_levels_ksp_chebyshev_esteig_steps", 50) solver_u =
PETScKrylovSolver("cg", pc)

I am encountering this error sometimes during the solution of the linear
system:
Error: Unable to successfully call PETSc function 'KSPSolve'. *** Reason:
PETSc error code is: 77 (Petsc has generated inconsistent data). *** Where:
This error was encountered inside
/tmp/dolfin/dolfin/la/PETScKrylovSolver.cpp.

*** Process: 130

Can anybody help me in understanding the possible reasons for this error
code? Or refer me to any available documentation? It will be really
helpful.

Also, I will like to know if there is a way to prevent the code from
stopping due to such PETSc errors, so that I can tweak some parameters to
hopefully make it progress.

Thanks and Regards,
Aditya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/bf2abbbe/attachment.html>

From jed at jedbrown.org  Fri Feb  8 11:49:38 2019
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 Feb 2019 10:49:38 -0700
Subject: [petsc-users] Meaning of PETSc error code 77
In-Reply-To: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
References: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
Message-ID: <87womakw5p.fsf@jedbrown.org>

Can you run in a debugger to get a stack trace?  I believe Dolfin
swallows the stack trace to obstruct efforts to remove bugs; probably
paid off by the bug lobby.

aditya kumar via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hello,
>
> I am using PETSc with FEniCS project libraries to solve a nonlinear
> problem. I am using PETSc Krylov solver with the following configuration
>
> pc = PETScPreconditioner("petsc_amg")
> PETScOptions.set("mg_levels_ksp_type", "chebyshev")
> PETScOptions.set("mg_levels_pc_type", "jacobi")
> PETScOptions.set("mg_levels_esteig_ksp_type", "cg")
> PETScOptions.set("mg_levels_ksp_chebyshev_esteig_steps", 50) solver_u =
> PETScKrylovSolver("cg", pc)
>
> I am encountering this error sometimes during the solution of the linear
> system:
> Error: Unable to successfully call PETSc function 'KSPSolve'. *** Reason:
> PETSc error code is: 77 (Petsc has generated inconsistent data). *** Where:
> This error was encountered inside
> /tmp/dolfin/dolfin/la/PETScKrylovSolver.cpp.
>
> *** Process: 130
>
> Can anybody help me in understanding the possible reasons for this error
> code? Or refer me to any available documentation? It will be really
> helpful.
>
> Also, I will like to know if there is a way to prevent the code from
> stopping due to such PETSc errors, so that I can tweak some parameters to
> hopefully make it progress.
>
> Thanks and Regards,
> Aditya

From jychang48 at gmail.com  Fri Feb  8 12:00:13 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 8 Feb 2019 11:00:13 -0700
Subject: [petsc-users] Meaning of PETSc error code 77
In-Reply-To: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
References: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
Message-ID: <CAP2=TMgs5PiVey9fFJokBu5BDnObgeF8QSDoSePWf+YZqbznGw@mail.gmail.com>

 Hi Aditya,

I wouldn't trust FEniCS' wrappers around PETSc. It might also depend on the
version of DOLFIN you're working with.

Shameless plug: Try using pfibs

https://github.com/NREL/pfibs

 We wrote a slightly better PETSc interface to FEniCS. It does require
FEniCS 2018.1.0 or higher. This should at least give better error logs for
your problem.

Also I know a lot of PETSc solvers rely on some sort of DM, which FEniCS
(at the moment) does not automatically provide. We have a pFibs branch
which attaches a DMShell to your functionspace.

https://github.com/NREL/pfibs/tree/dm-nested-solver-feature

It's still in its early stages, but it should at least be able to handle
the PETScOptions you've set above.

Hopefully this helps,
Justin

On Fri, Feb 8, 2019 at 10:41 AM aditya kumar via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I am using PETSc with FEniCS project libraries to solve a nonlinear
> problem. I am using PETSc Krylov solver with the following configuration
>
> pc = PETScPreconditioner("petsc_amg")
> PETScOptions.set("mg_levels_ksp_type", "chebyshev")
> PETScOptions.set("mg_levels_pc_type", "jacobi")
> PETScOptions.set("mg_levels_esteig_ksp_type", "cg")
> PETScOptions.set("mg_levels_ksp_chebyshev_esteig_steps", 50) solver_u =
> PETScKrylovSolver("cg", pc)
>
> I am encountering this error sometimes during the solution of the linear
> system:
> Error: Unable to successfully call PETSc function 'KSPSolve'. *** Reason:
> PETSc error code is: 77 (Petsc has generated inconsistent data). *** Where:
> This error was encountered inside
> /tmp/dolfin/dolfin/la/PETScKrylovSolver.cpp.
>
> *** Process: 130
>
> Can anybody help me in understanding the possible reasons for this error
> code? Or refer me to any available documentation? It will be really
> helpful.
>
> Also, I will like to know if there is a way to prevent the code from
> stopping due to such PETSc errors, so that I can tweak some parameters to
> hopefully make it progress.
>
> Thanks and Regards,
> Aditya
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/36f48832/attachment.html>

From stefano.zampini at gmail.com  Fri Feb  8 12:21:29 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Fri, 8 Feb 2019 21:21:29 +0300
Subject: [petsc-users] Meaning of PETSc error code 77
In-Reply-To: <87womakw5p.fsf@jedbrown.org>
References: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
	<87womakw5p.fsf@jedbrown.org>
Message-ID: <CAGPUisjEHJz5rDF98o_ZCbn52ddw5+eGAxjuXMBEeA41JdGz0g@mail.gmail.com>

Dolfin pushes its own error handler, to pretty print errors from petsc,
leaving the user clueless about what happened. To get the stack trace, you
can also call PetscPopErrorHandler before you call the costructor of the
krylov solver

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscPopErrorHandler.html

Il giorno Ven 8 Feb 2019, 20:49 Jed Brown via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

> Can you run in a debugger to get a stack trace?  I believe Dolfin
> swallows the stack trace to obstruct efforts to remove bugs; probably
> paid off by the bug lobby.
>
> aditya kumar via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Hello,
> >
> > I am using PETSc with FEniCS project libraries to solve a nonlinear
> > problem. I am using PETSc Krylov solver with the following configuration
> >
> > pc = PETScPreconditioner("petsc_amg")
> > PETScOptions.set("mg_levels_ksp_type", "chebyshev")
> > PETScOptions.set("mg_levels_pc_type", "jacobi")
> > PETScOptions.set("mg_levels_esteig_ksp_type", "cg")
> > PETScOptions.set("mg_levels_ksp_chebyshev_esteig_steps", 50) solver_u =
> > PETScKrylovSolver("cg", pc)
> >
> > I am encountering this error sometimes during the solution of the linear
> > system:
> > Error: Unable to successfully call PETSc function 'KSPSolve'. *** Reason:
> > PETSc error code is: 77 (Petsc has generated inconsistent data). ***
> Where:
> > This error was encountered inside
> > /tmp/dolfin/dolfin/la/PETScKrylovSolver.cpp.
> >
> > *** Process: 130
> >
> > Can anybody help me in understanding the possible reasons for this error
> > code? Or refer me to any available documentation? It will be really
> > helpful.
> >
> > Also, I will like to know if there is a way to prevent the code from
> > stopping due to such PETSc errors, so that I can tweak some parameters to
> > hopefully make it progress.
> >
> > Thanks and Regards,
> > Aditya
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/f7f017b5/attachment.html>

From jychang48 at gmail.com  Fri Feb  8 16:56:57 2019
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 8 Feb 2019 15:56:57 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
	<DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>
Message-ID: <CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>

So I used -mat_view draw -draw_pause -1 on my medium sized matrix and got
this output:

[image: 1MPI.png]

So it seems there are lots of off-diagonal terms, and that a decomposition
of the problem via matload would give a terrible unbalanced problem.

Given the initial A and b Mat/Vec, I experimented with MatPartioning and
inserted the following lines into my code:

Mat               Apart;
Vec               bpart;
MatPartitioning   part;
IS                is,isrows;
ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part);CHKERRQ(ierr);
ierr = MatPartitioningSetAdjacency(part, A);CHKERRQ(ierr);
ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
ierr = MatPartitioningApply(part, &is);CHKERRQ(ierr);
ierr = ISBuildTwoSided(is,NULL,&isrows);CHKERRQ(ierr);
ierr = MatCreateSubMatrix(A, isrows,isrows, MAT_INITIAL_MATRIX,
&Apart);CHKERRQ(ierr);
ierr = MatSetOptionsPrefix(Apart, "part_");CHKERRQ(ierr);
ierr = MatSetFromOptions(Apart);CHKERRQ(ierr);
ierr = VecGetSubVector(b,isrows,&bpart);CHKERRQ(ierr);

/* Set Apart and bpart in the KSPSolve */
...

And here are the mat_draw figures from 2 and 4 MPI processes respectively:

[image: 2MPI.png][image: 4MPI.png]

Is this "right"? It just feels like I'm just duplicating the nnz structure
among all the MPI processes. And it didn't really improve the performance
of ASM.

Also, this nnz structure appears to depend on the distribution system we
cook up in OpenDSS, for example the largest matrix we have thus far looks
like this:
[image: larger.png]
Is there a better alternative to the partitioning I implemented above? Or
did I do something catastrophically wrong

Thanks,
Justin

Side node - Yes I tried KLU, but it needed several KSP iterations to obtain
the solution. Here's the ksp monitor/view outputs:

0 KSP preconditioned resid norm 1.705434112839e+06 true resid norm
2.242813827253e+12 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.122965789284e+05 true resid norm
3.057749589444e+11 ||r(i)||/||b|| 1.363354172463e-01
  2 KSP preconditioned resid norm 1.962518730076e+04 true resid norm
2.510054932552e+10 ||r(i)||/||b|| 1.119154386357e-02
  3 KSP preconditioned resid norm 3.094963519133e+03 true resid norm
6.489763653495e+09 ||r(i)||/||b|| 2.893581078660e-03
  4 KSP preconditioned resid norm 1.755871992454e+03 true resid norm
2.315676037474e+09 ||r(i)||/||b|| 1.032486963178e-03
  5 KSP preconditioned resid norm 1.348939340771e+03 true resid norm
1.864929933344e+09 ||r(i)||/||b|| 8.315134812722e-04
  6 KSP preconditioned resid norm 5.532203694243e+02 true resid norm
9.985525631209e+08 ||r(i)||/||b|| 4.452231170449e-04
  7 KSP preconditioned resid norm 3.636087020506e+02 true resid norm
5.712899201028e+08 ||r(i)||/||b|| 2.547201703329e-04
  8 KSP preconditioned resid norm 2.926812321412e+02 true resid norm
3.627282296417e+08 ||r(i)||/||b|| 1.617290856843e-04
  9 KSP preconditioned resid norm 1.629184033135e+02 true resid norm
1.048838851435e+08 ||r(i)||/||b|| 4.676441881580e-05
 10 KSP preconditioned resid norm 8.297821067807e+01 true resid norm
3.423640694920e+07 ||r(i)||/||b|| 1.526493484800e-05
 11 KSP preconditioned resid norm 2.997246200648e+01 true resid norm
9.880250538293e+06 ||r(i)||/||b|| 4.405292324417e-06
 12 KSP preconditioned resid norm 2.156940809471e+01 true resid norm
3.521518932572e+06 ||r(i)||/||b|| 1.570134306192e-06
 13 KSP preconditioned resid norm 1.211823308437e+01 true resid norm
1.130205443318e+06 ||r(i)||/||b|| 5.039229870908e-07
Linear solve converged due to CONVERGED_RTOL iterations 13
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=50, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: lu
    out-of-place factorization
    tolerance for zero pivot 2.22045e-14
    matrix ordering: amd
    factor fill ratio given 0., needed 0.
      Factored matrix follows:
        Mat Object: 1 MPI processes
          type: klu
          rows=320745, cols=320745
          package used to perform factorization: klu
          total: nonzeros=0, allocated nonzeros=0
          total number of mallocs used during MatSetValues calls =0
            KLU stats:
              Number of diagonal blocks: 1
              Total nonzeros=2426908
            KLU runtime parameters:
              Partial pivoting tolerance: 0.001
              BTF preordering enabled: 1
              Ordering: AMD (not using the PETSc ordering)
              Matrix row scaling: NONE
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
    type: seqaij
    rows=320745, cols=320745
    total: nonzeros=1928617, allocated nonzeros=1928617
    total number of mallocs used during MatSetValues calls =0
      using I-node routines: found 186383 nodes, limit used is 5

It was slower than the PETSc LU solver which doesn't seem to make sense to
me. The lines:

 total: nonzeros=0, allocated nonzeros=0
 total number of mallocs used during MatSetValues calls =0

seem to indicate that KLU needs to be customized to get the "high
performance" that the circuits community claim to have?

On Thu, Feb 7, 2019 at 8:03 AM Abhyankar, Shrirang G <
shrirang.abhyankar at pnnl.gov> wrote:

> Hi Justin,
>
>     The serial solve time for this matrix of size 320K is about 0.3 secs,
> which I think is pretty fast. As such, I am not surprised you are not
> getting any speed up in parallel with asm. Perhaps, you need a bigger
> matrix?
>
>
>
> It would be interesting to see a spy plot of the matrix on different
> processors. If there are a lot of off-diagonal elements, then asm may not
> work that well. You may need to use a partitioner in that case.
>
>
>
> As a side note, folks in the circuits community consider KLU as one of the
> fastest serial solvers for such kind of problems. You can try that out and
> see if it improves serial performance even further.
>
>
>
> --download-suitesparse
>
>
>
> -pc_factor_mat_solver_package klu -pc_factor_mat_ordering_type amd
>
>
>
> Shri
>
>
>
> *From:* Justin Chang <jychang48 at gmail.com>
> *Sent:* Wednesday, February 6, 2019 2:56 PM
> *To:* Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
> *Cc:* Mark Adams <mfadams at lbl.gov>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Preconditioning systems of equations with
> complex numbers
>
>
>
> Hi Shri,
>
>
>
> Yeah so here at NREL I'm working with a group of folks (specifically
> Dheepak Krishnamurthy, who apparently knows you back when y'all were both
> at Argonne) and asked them to cook up a urban-suburban distribution system
> from OpenDSS (he wrote a Julia/Python wrapper for OpenDSS). I used the
> following options:
>
>
> -ksp_type gmres
> -pc_type asm
> -pc_asm_overlap 3
> -sub_pc_type lu
> -sub_pc_factor_mat_ordering_type amd # using rcm also produced similar
> results, all of which better than the default
>
> -ksp_monitor_true_residual
>
> -ksp_view
>
> -log_view
>
>
>
> And here are the long overdue outputs for up to 16 MPI processes on our
> Eagle HPC system (with Skylake server nodes).
>
>
>
> Basically, the algorithmic convergence degrades when more MPI procs are
> used, and while I could ameliorate the KSP count by increasing the overlap,
> it slowdowns the overall computation which I assume is from the
> communication latencies. Also note that the KSPSolve time also slowed down
> because of the ksp_monitor
>
>
>
> Is this a "big performance hit"? I don't know, maybe I need to first start
> off with a fully 3-phase system and see if similar solver characteristics
> are found?
>
>
>
> Thanks,
>
> Justin
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Feb 5, 2019 at 6:09 PM Abhyankar, Shrirang G <
> shrirang.abhyankar at pnnl.gov> wrote:
>
> Hi Justin,
>
>    Typically, the power grid distribution systems have a radial structure
> (unless it is an urban area) that leads to a, more or less, staircase type
> matrix. So a MatLoad() or VecLoad() would presumably just just splits the
> stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the
> phases for a bus to be split across processors, but I don?t think it would
> cause a big performance hit.
>
> I?ve worked with OpenDSS recently and have created a C wrapper for it that
> you can use to query values from it. I can help out with your
> experimentations, if you need.
>
>
>
> Shri
>
> *From:* petsc-users <petsc-users-bounces at mcs.anl.gov> *On Behalf Of *Justin
> Chang via petsc-users
> *Sent:* Tuesday, February 5, 2019 5:23 PM
> *To:* Mark Adams <mfadams at lbl.gov>
> *Cc:* petsc-users <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Preconditioning systems of equations with
> complex numbers
>
>
>
> Hi all,
>
>
>
> So I *think* I know what the problem is. I am modeling a a distribution
> system where each bus can have anywhere from 1 to 3 phases. In the smallest
> example I had, all four buses had 3 phases (hence 12 dofs), and this was
> very easy to decompose - each MPI process should have a specific set of
> complex voltages on their respective nodes/buses. The larger test cases
> have lots of single phase buses attached, so a naive decomposition of the
> matrix isn't trivial. Again we're only extracting the Y-bus matrices out of
> an external software (OpenDSS to be specific) so we have no a priori
> information on how they organize their degrees-of-freedom - we let
> MatLoad() and VecLoad() do the decomposition. So it's possible that during
> this decomposition, a single bus' phases are being split across two or more
> MPI processes, which I assume would mess up the algorithmic performance of
> GAMG/ASM when more than one MPI process is needed.
>
>
>
> Yeah DMNetwork will fix this and ensure that all dofs per vertex do not
> get split across different MPI procs, but for now we're just trying to see
> if we can get a simple proof-of-concept working. We're trying to come up
> with a toy distribution system where everything is three-phase (hence very
> straightforward to decompose even with MatLoad().
>
>
>
> Justin
>
>
>
> On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov> wrote:
>
> I would stay away from eigen estimates in the solver (but give us the
> spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.
>
>
>
> Applications that have lived on direct solvers can add sorts of crap like
> penalty terms.
>
>
>
> sor seemed to work OK so I'd check the coarse grids in GAMG. Test with
> just two levels. That way you don't have to use sor on an internal MG grid,
> which can be bad and probably is here.
>
>
>
>
>
> On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>
>
> > On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Thanks everyone for your suggestions/feedback. So a few things:
> >
> > 1) When I examined larger distribution networks (~150k buses) my
> eigenvalue estimates from the chebyshev method get enormous indeed. See
> below:
> >
> > [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1,
> nnz/row (ave)=6, np=1
> > [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with
> threshold 0., 6.01293 nnz ave. (N=320745)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00
> min=4.639523e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with
> threshold 0., 7.32435 nnz ave. (N=44797)
> > [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00
> min=9.687506e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1
> active pes
> > [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with
> threshold 0., 22.5607 nnz ave. (N=13043)
> > [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00
> min=6.150856e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active
> pes
> > [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with
> threshold 0., 79.5142 nnz ave. (N=2256)
> > [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00
> min=6.780909e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with
> threshold 0., 206.379 nnz ave. (N=232)
> > [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00
> min=2.153627e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
> >
> > 2) I tried all the suggestions mentioned before: setting
> -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my
> convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more
> iterations of richardson/sor.
> >
> > 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just
> solving LU on the whole system.
> >
> > 3b)The problem is that I can't get it to even speedup across 2 MPI
> processes for my 150k bus case (~300k dofs) - and I already checked that
> this problem could in theory be parallelized by setting the -ksp_max_it to
> something low and observing that the KSPSolve time decreases as MPI
> concurrency increases. The potential speedup is countered by the fact that
> the algorithmic convergence rate blows up when either more MPI processes
> are added or when I tune the block_size/overlap parameters.
>
>    Yes, we just haven't gotten a scalable preconditioner yet for your
> matrix.
>
> >
> > Which leads me to one of two conclusions/questions:
> >
> > A) Is my 300 DOF problem still too small? Do I need to look at a problem
> with, say, 10 Million DOF or more to see that increasing the asm_block_size
> will have better performance (despite having more KSP iterations) than a
> single giant asm_block?
>
>    No, if you had a scalable preconditioner you would see good speed up
> with 300k unknowns, using 10 million won't help.
>
> >
> > B) Or is there something even more sinister about my system of equations
> in complex form?
>
>   There is a technical term for matrices for which both GAMG and ASM do
> very poorly on: nasty :-)
>
>   Have you tried using parallel LU (for example ./configure
> --download-superlu_dist). Direct LU is the last refuge of the
> unpreconditionable. How large will your matrices get?
>
>    Barry
>
> >
> > Thanks,
> > Justin
> >
> > On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > The coarse grid is getting set to SOR rather than LU.
> >
> >    Matt
> >
> > On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
> > I tried these options:
> >
> > -ksp_view
> > -ksp_monitor_true_residual
> > -ksp_type gmres
> > -ksp_max_it 50
> > -pc_type gamg
> > -mg_coarse_pc_type sor
> > -mg_levels_1_ksp_type richardson
> > -mg_levels_2_ksp_type richardson
> > -mg_levels_3_ksp_type richardson
> > -mg_levels_4_ksp_type richardson
> > -mg_levels_1_pc_type sor
> > -mg_levels_2_pc_type sor
> > -mg_levels_3_pc_type sor
> > -mg_levels_4_pc_type sor
> >
> > And still have a non-converging solution:
> >
> >   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm
> 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm
> 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
> >   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm
> 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
> >   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm
> 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
> >   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm
> 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
> >   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm
> 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
> >   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm
> 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
> >   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm
> 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
> >   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm
> 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
> >   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm
> 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> >  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm
> 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> >  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm
> 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> >  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm
> 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> >  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm
> 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> >  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm
> 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> >  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm
> 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> >  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm
> 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> >  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm
> 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> >  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm
> 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> >  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm
> 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> >  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm
> 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> >  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm
> 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> >  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm
> 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> >  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm
> 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> >  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm
> 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> >  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm
> 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> >  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm
> 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> >  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm
> 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> >  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm
> 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> >  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm
> 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> >  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm
> 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> >  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm
> 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> >  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm
> 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> >  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm
> 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> >  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm
> 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> >  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm
> 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> >  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm
> 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> >  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm
> 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> >  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm
> 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> >  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm
> 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> >  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm
> 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> >  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm
> 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> >  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm
> 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> >  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm
> 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> >  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm
> 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> >  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm
> 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> >  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm
> 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> >  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm
> 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> >  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm
> 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> >  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm
> 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> >  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm
> 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> > KSP Object: 1 MPI processes
> >   type: gmres
> >     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >     happy breakdown tolerance 1e-30
> >   maximum iterations=50, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: gamg
> >     type is MULTIPLICATIVE, levels=5 cycles=v
> >       Cycles per PCApply=1
> >       Using externally compute Galerkin coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values in graph on each level =
>  0.   0.   0.
> >         Threshold scaling factor for each level not specified = 1.
> >         AGG specific options
> >           Symmetric graph false
> >           Number of levels to square graph 1
> >           Number smoothing steps 1
> >         Complexity:    grid = 1.31821
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object: (mg_coarse_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_coarse_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=17, cols=17
> >         total: nonzeros=163, allocated nonzeros=163
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 12 nodes, limit used is 5
> >   Down solver (pre-smoother) on level 1 -------------------------------
> >     KSP Object: (mg_levels_1_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_1_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=100, cols=100
> >         total: nonzeros=1240, allocated nonzeros=1240
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 2 -------------------------------
> >     KSP Object: (mg_levels_2_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_2_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=537, cols=537
> >         total: nonzeros=5291, allocated nonzeros=5291
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 3 -------------------------------
> >     KSP Object: (mg_levels_3_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_3_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=1541, cols=1541
> >         total: nonzeros=8039, allocated nonzeros=8039
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 4 -------------------------------
> >     KSP Object: (mg_levels_4_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_4_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=8541, cols=8541
> >         total: nonzeros=46299, allocated nonzeros=46299
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 5464 nodes, limit used is 5
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> >     type: seqaij
> >     rows=8541, cols=8541
> >     total: nonzeros=46299, allocated nonzeros=46299
> >     total number of mallocs used during MatSetValues calls =0
> >       using I-node routines: found 5464 nodes, limit used is 5
> >
> > Am I doing this right? Did I miss anything?
> >
> >
> >
> > On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
> > Hi Mark,
> >
> > 1) So with these options:
> >
> > -ksp_type gmres
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -ksp_converged_reason
> > -pc_type bjacobi
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm
> 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
> >   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm
> 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
> >   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm
> 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
> >   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm
> 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
> >   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm
> 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 5
> >
> > 2) With richardson/sor:
> >
> > Okay, its looks like Richardson/SOR solves this just fine. You can use
> this as the smoother for GAMG instead
> > of Cheby/Jacobi, and probably see better results on the larger problems.
> >
> >   Matt
> >
> > -ksp_type richardson
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type sor
> >
> > This is what I get:
> >
> >   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm
> 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm
> 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
> >   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm
> 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
> >   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm
> 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
> >   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm
> 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
> >   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm
> 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
> >   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm
> 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
> >   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm
> 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
> >   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm
> 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
> >   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm
> 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
> >  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm
> 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
> >  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm
> 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
> >  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm
> 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
> >  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm
> 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
> >  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm
> 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
> >  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm
> 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
> >  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm
> 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
> >  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm
> 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
> >  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm
> 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
> >  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm
> 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
> >  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm
> 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
> >  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm
> 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
> >  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm
> 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
> >  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm
> 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
> >  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm
> 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
> >  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm
> 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
> >  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm
> 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
> >  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm
> 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
> >  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm
> 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
> >  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm
> 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
> >  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm
> 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
> >  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm
> 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
> >  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm
> 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
> >  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm
> 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
> >  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm
> 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
> >  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm
> 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
> >  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm
> 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
> >  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm
> 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
> >  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm
> 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
> >  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm
> 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
> >  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm
> 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 40
> >
> > 3) And lastly with chebyshev/jacobi:
> >
> > -ksp_type chebyshev
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type jacobi
> >
> >   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm
> 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
> >   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm
> 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
> >   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm
> 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
> >   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm
> 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
> >   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm
> 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
> >   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm
> 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
> >   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm
> 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
> >   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm
> 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
> >   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm
> 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
> >   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm
> 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
> >  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm
> 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
> >  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm
> 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
> >  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm
> 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
> >  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm
> 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
> >  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm
> 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
> >  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm
> 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
> >  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm
> 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
> >  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm
> 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
> >  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm
> 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
> >  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm
> 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
> >  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm
> 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
> >  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm
> 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
> >  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm
> 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
> >  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm
> 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
> >  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm
> 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
> >  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm
> 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
> >  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm
> 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
> >  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm
> 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
> >  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm
> 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
> >  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm
> 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
> >  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm
> 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
> >  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm
> 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
> >  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm
> 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
> >  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm
> 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
> >  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm
> 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
> >  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm
> 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
> >  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm
> 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
> >  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm
> 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
> >  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm
> 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
> >  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm
> 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
> >  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm
> 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
> >  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm
> 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
> >  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm
> 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
> >  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm
> 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
> >  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm
> 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
> >  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm
> 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
> >  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm
> 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
> >  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm
> 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
> >  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm
> 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
> >  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm
> 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
> >  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm
> 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
> >  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm
> 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
> >  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm
> 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
> >  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm
> 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
> >  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm
> 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
> >  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm
> 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
> >  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm
> 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
> >  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm
> 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
> >  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm
> 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
> >  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm
> 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
> >  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm
> 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
> >  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm
> 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
> >  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm
> 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
> >  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm
> 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
> >  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm
> 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
> >  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm
> 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
> >  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm
> 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 67
> >
> > Looks like neither of these ksp/pc combos are good? I also tried
> -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the
> GAMG info I was able to grep from the out-of-box params:
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00
> min=2.394056e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 5.21674 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00
> min=6.783380e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 9.85289 nnz ave. (N=537)
> > [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00
> min=5.974776e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 12.4 nnz ave. (N=100)
> > [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00
> min=4.842076e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> >
> > And the one with pc_gamg_agg_nsmooths 0
> >
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1,
> nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with
> threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with
> threshold 0., 3.07398 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> > [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.02211 nnz ave. (N=814)
> > [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> > [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3.00434 nnz ave. (N=461)
> > [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> > [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=290)
> > [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> > [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 3. nnz ave. (N=197)
> > [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> > [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active
> pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.98425 nnz ave. (N=127)
> > [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> > [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.97561 nnz ave. (N=82)
> > [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> > [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold
> 0., 2.9697 nnz ave. (N=66)
> > [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> > [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> >
> >
> > Thanks,
> > Justin
> >
> > On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> >
> >
> > Both GAMG and ILU are nice and dandy for this,
> >
> > I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and
> converge it way down, say rtol = 1.e-12. See which one is better in the
> early iteration and pick it. It would be nice to check that it solves the
> problem ...
> >
> > The residual drops about 5 orders in the first iteration and then
> flatlines. That is very bad. Check that the smoother can actually solve the
> problem.
> >
> > but as soon as I look at a bigger system, like a network with 8500
> buses, the out-of-box gamg craps out. I am not sure where to start when it
> comes to tuning GAMG.
> >
> > First try -pc_gamg_nsmooths 0
> >
> > You can run with -info and grep on GAMG to see some diagnostic output.
> Eigen estimates are fragile in practice and with this parameter and SOR
> there are no eigen estimates needed. The max eigen values for all levels
> should be between say 2 (or less) and say 3-4. Much higher is a sign of a
> problem.
> >
> > Attached is the ksp monitor/view output for gamg on the unsuccessful
> solve
> >
> > I'm also attaching a zip file which contains the simple PETSc script
> that loads the binary matrix/vector as well as two test cases, if you guys
> are interested in trying it out. It only works if you have PETSc configured
> with complex numbers.
> >
> > Thanks
> >
> > Justin
> >
> > PS - A couple years ago I had asked if there was a paper/tutorial on
> using/tuning GAMG. Does such a thing exist today?
> >
> > There is a write up in the manual that is tutorial like.
> >
> >
> > On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com>
> wrote:
> > Here's IMHO the simplest explanation of the equations I'm trying to
> solve:
> >
> > http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> >
> > Right now we're just trying to solve eq(5) (in section 1), inverting the
> linear Y-bus matrix. Eventually we have to be able to solve equations like
> those in the next section.
> >
> > Maybe I am reading this wrong, but the Y-bus matrix looks like an
> M-matrix to me (if all the y's are positive). This means
> > that it should be really easy to solve, and I think GAMG should do it.
> You can start out just doing relaxation, like SOR, on
> > small examples.
> >
> >   Thanks,
> >
> >     Matt
> >
> > On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> > Hi all,
> >
> > I'm working with some folks to extract a linear system of equations from
> an external software package that solves power flow equations in complex
> form. Since that external package uses serial direct solvers like KLU from
> suitesparse, I want a proof-of-concept where the same matrix can be solved
> in PETSc using its parallel solvers.
> >
> > I got mumps to achieve a very minor speedup across two MPI processes on
> a single node (went from solving a 300k dog system in 1.8 seconds to 1.5
> seconds). However I want to use iterative solvers and preconditioners but I
> have never worked with complex numbers so I am not sure what the "best"
> options are given PETSc's capabilities.
> >
> > So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I
> believe I also tried BICG with BJACOBI and while it did converge it
> converged slowly. Does anyone have recommendations on how one would go
> about preconditioning PETSc matrices with complex numbers? I was originally
> thinking about converting it to cartesian form: Declaring all voltages =
> sqrt(real^2+imaginary^2) and all angles to be something like a conditional
> arctan(imaginary/real) because all the papers I've seen in literature that
> claim to successfully precondition power flow equations operate in this
> form.
> >
> > 1) We really need to see the (simplified) equations
> >
> > 2) All complex equations can be converted to a system of real equations
> twice as large, but this is not necessarily the best way to go
> >
> >  Thanks,
> >
> >     Matt
> >
> > Justin
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/1daeb100/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1MPI.png
Type: image/png
Size: 25884 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/1daeb100/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2MPI.png
Type: image/png
Size: 19966 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/1daeb100/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 4MPI.png
Type: image/png
Size: 15018 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/1daeb100/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: larger.png
Type: image/png
Size: 18636 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190208/1daeb100/attachment-0007.png>

From bsmith at mcs.anl.gov  Sat Feb  9 01:49:01 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 9 Feb 2019 07:49:01 +0000
Subject: [petsc-users] Meaning of PETSc error code 77
In-Reply-To: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
References: <CALpwOdmVLVrjaaArY=TUyszQyVivN+3FP6tLe65=AA4DepZdwQ@mail.gmail.com>
Message-ID: <02B8B47B-8E49-4CD5-8417-80FAA10A252E@anl.gov>


   This particular error "Petsc has generated inconsistent data" is a catastrophic error (not, for example, simply failure to solve a system) so it is not possible to by-pass the error. We would need to know exactly where in the PETSc code the error was detected to to help you resolve the problem.

   Barry


> On Feb 8, 2019, at 11:40 AM, aditya kumar via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
> 
> I am using PETSc with FEniCS project libraries to solve a nonlinear problem. I am using PETSc Krylov solver with the following configuration
> 
> pc = PETScPreconditioner("petsc_amg") 
> PETScOptions.set("mg_levels_ksp_type", "chebyshev")
> PETScOptions.set("mg_levels_pc_type", "jacobi")
> PETScOptions.set("mg_levels_esteig_ksp_type", "cg")
> PETScOptions.set("mg_levels_ksp_chebyshev_esteig_steps", 50)
> solver_u = PETScKrylovSolver("cg", pc)
> 
> 
> 
> I am encountering this error sometimes during the solution of the linear system:
> 
> Error:   Unable to successfully call PETSc function 'KSPSolve'.
> *** Reason:  PETSc error code is: 77 (Petsc has generated inconsistent data).
> *** Where:   This error was encountered inside /tmp/dolfin/dolfin/la/PETScKrylovSolver.cpp. 
> 
> *** Process: 130
> 
> Can anybody help me in understanding the possible reasons for this error code? Or refer me to any available documentation? It will be really helpful. 
> 
> Also, I will like to know if there is a way to prevent the code from stopping due to such PETSc errors, so that I can tweak some parameters to hopefully make it progress.  
> 
> 
> Thanks and Regards,
> Aditya
> 


From shrirang.abhyankar at pnnl.gov  Mon Feb 11 08:41:17 2019
From: shrirang.abhyankar at pnnl.gov (Abhyankar, Shrirang G)
Date: Mon, 11 Feb 2019 14:41:17 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
	<DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>
Message-ID: <DM6PR09MB3244E1F15AC69972794BCD95E7640@DM6PR09MB3244.namprd09.prod.outlook.com>



From: Justin Chang <jychang48 at gmail.com>
Sent: Friday, February 8, 2019 4:57 PM
To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
Cc: Mark Adams <mfadams at lbl.gov>; PETSc users list <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

So I used -mat_view draw -draw_pause -1 on my medium sized matrix and got this output:

[1MPI.png]

So it seems there are lots of off-diagonal terms, and that a decomposition of the problem via matload would give a terrible unbalanced problem.

Given the initial A and b Mat/Vec, I experimented with MatPartioning and inserted the following lines into my code:

Mat               Apart;
Vec               bpart;
MatPartitioning   part;
IS                is,isrows;
ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part);CHKERRQ(ierr);
ierr = MatPartitioningSetAdjacency(part, A);CHKERRQ(ierr);
ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
ierr = MatPartitioningApply(part, &is);CHKERRQ(ierr);
ierr = ISBuildTwoSided(is,NULL,&isrows);CHKERRQ(ierr);
ierr = MatCreateSubMatrix(A, isrows,isrows, MAT_INITIAL_MATRIX, &Apart);CHKERRQ(ierr);
ierr = MatSetOptionsPrefix(Apart, "part_");CHKERRQ(ierr);
ierr = MatSetFromOptions(Apart);CHKERRQ(ierr);
ierr = VecGetSubVector(b,isrows,&bpart);CHKERRQ(ierr);

/* Set Apart and bpart in the KSPSolve */
...

And here are the mat_draw figures from 2 and 4 MPI processes respectively:

[2MPI.png][4MPI.png]

Is this "right"? It just feels like I'm just duplicating the nnz structure among all the MPI processes. And it didn't really improve the performance of ASM.

This looks kind of right. You have the big fat diagonal blocks represent separate meshed networks, and the sparse off-diagonal blocks that show very less connectivity between the networks. This is typical of power grid where the connectivity between different areas is sparse. However, the blocks being identical is puzzling. I would expect the blocks to be of different sizes. Was this network created by duplicating smaller networks?
I was hoping ASM would be faster on the partitioned matrix rather than the original one. What options are you using for sub_pc? -sub_pc_type lu -sub_pc_factor_mat_ordering_type amd?

Also, this nnz structure appears to depend on the distribution system we cook up in OpenDSS, for example the largest matrix we have thus far looks like this:
[larger.png]
Yes, the matrix structure depends on the network structure. This to me looks like a star-shaped network where you have a tightly interconnected network (bottom dense block) and smallish radial distribution networks emanating from the nodes. That?s just one guess. It could be some other type as well.


Is there a better alternative to the partitioning I implemented above? Or did I do something catastrophically wrong

Thanks,
Justin

Side node - Yes I tried KLU, but it needed several KSP iterations to obtain the solution. Here's the ksp monitor/view outputs:

0 KSP preconditioned resid norm 1.705434112839e+06 true resid norm 2.242813827253e+12 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.122965789284e+05 true resid norm 3.057749589444e+11 ||r(i)||/||b|| 1.363354172463e-01
  2 KSP preconditioned resid norm 1.962518730076e+04 true resid norm 2.510054932552e+10 ||r(i)||/||b|| 1.119154386357e-02
  3 KSP preconditioned resid norm 3.094963519133e+03 true resid norm 6.489763653495e+09 ||r(i)||/||b|| 2.893581078660e-03
  4 KSP preconditioned resid norm 1.755871992454e+03 true resid norm 2.315676037474e+09 ||r(i)||/||b|| 1.032486963178e-03
  5 KSP preconditioned resid norm 1.348939340771e+03 true resid norm 1.864929933344e+09 ||r(i)||/||b|| 8.315134812722e-04
  6 KSP preconditioned resid norm 5.532203694243e+02 true resid norm 9.985525631209e+08 ||r(i)||/||b|| 4.452231170449e-04
  7 KSP preconditioned resid norm 3.636087020506e+02 true resid norm 5.712899201028e+08 ||r(i)||/||b|| 2.547201703329e-04
  8 KSP preconditioned resid norm 2.926812321412e+02 true resid norm 3.627282296417e+08 ||r(i)||/||b|| 1.617290856843e-04
  9 KSP preconditioned resid norm 1.629184033135e+02 true resid norm 1.048838851435e+08 ||r(i)||/||b|| 4.676441881580e-05
 10 KSP preconditioned resid norm 8.297821067807e+01 true resid norm 3.423640694920e+07 ||r(i)||/||b|| 1.526493484800e-05
 11 KSP preconditioned resid norm 2.997246200648e+01 true resid norm 9.880250538293e+06 ||r(i)||/||b|| 4.405292324417e-06
 12 KSP preconditioned resid norm 2.156940809471e+01 true resid norm 3.521518932572e+06 ||r(i)||/||b|| 1.570134306192e-06
 13 KSP preconditioned resid norm 1.211823308437e+01 true resid norm 1.130205443318e+06 ||r(i)||/||b|| 5.039229870908e-07
Linear solve converged due to CONVERGED_RTOL iterations 13
KSP Object: 1 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=50, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: lu
    out-of-place factorization
    tolerance for zero pivot 2.22045e-14
    matrix ordering: amd
    factor fill ratio given 0., needed 0.
      Factored matrix follows:
        Mat Object: 1 MPI processes
          type: klu
          rows=320745, cols=320745
          package used to perform factorization: klu
          total: nonzeros=0, allocated nonzeros=0
          total number of mallocs used during MatSetValues calls =0
            KLU stats:
              Number of diagonal blocks: 1
              Total nonzeros=2426908
            KLU runtime parameters:
              Partial pivoting tolerance: 0.001
              BTF preordering enabled: 1
              Ordering: AMD (not using the PETSc ordering)
              Matrix row scaling: NONE
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
    type: seqaij
    rows=320745, cols=320745
    total: nonzeros=1928617, allocated nonzeros=1928617
    total number of mallocs used during MatSetValues calls =0
      using I-node routines: found 186383 nodes, limit used is 5

It was slower than the PETSc LU solver which doesn't seem to make sense to me. The lines:

Hmm?that is strange. KLU is a direct solver so it should be as fast as PETSc?s direct solver. Can you send me the matrix and the code (offline). I?d like to test.

Shri
 total: nonzeros=0, allocated nonzeros=0
 total number of mallocs used during MatSetValues calls =0

seem to indicate that KLU needs to be customized to get the "high performance" that the circuits community claim to have?

On Thu, Feb 7, 2019 at 8:03 AM Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov<mailto:shrirang.abhyankar at pnnl.gov>> wrote:
Hi Justin,
    The serial solve time for this matrix of size 320K is about 0.3 secs, which I think is pretty fast. As such, I am not surprised you are not getting any speed up in parallel with asm. Perhaps, you need a bigger matrix?

It would be interesting to see a spy plot of the matrix on different processors. If there are a lot of off-diagonal elements, then asm may not work that well. You may need to use a partitioner in that case.

As a side note, folks in the circuits community consider KLU as one of the fastest serial solvers for such kind of problems. You can try that out and see if it improves serial performance even further.

--download-suitesparse

-pc_factor_mat_solver_package klu -pc_factor_mat_ordering_type amd

Shri

From: Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>>
Sent: Wednesday, February 6, 2019 2:56 PM
To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov<mailto:shrirang.abhyankar at pnnl.gov>>
Cc: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>; PETSc users list <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

Hi Shri,

Yeah so here at NREL I'm working with a group of folks (specifically Dheepak Krishnamurthy, who apparently knows you back when y'all were both at Argonne) and asked them to cook up a urban-suburban distribution system from OpenDSS (he wrote a Julia/Python wrapper for OpenDSS). I used the following options:

-ksp_type gmres
-pc_type asm
-pc_asm_overlap 3
-sub_pc_type lu
-sub_pc_factor_mat_ordering_type amd # using rcm also produced similar results, all of which better than the default
-ksp_monitor_true_residual
-ksp_view
-log_view

And here are the long overdue outputs for up to 16 MPI processes on our Eagle HPC system (with Skylake server nodes).

Basically, the algorithmic convergence degrades when more MPI procs are used, and while I could ameliorate the KSP count by increasing the overlap, it slowdowns the overall computation which I assume is from the communication latencies. Also note that the KSPSolve time also slowed down because of the ksp_monitor

Is this a "big performance hit"? I don't know, maybe I need to first start off with a fully 3-phase system and see if similar solver characteristics are found?

Thanks,
Justin





On Tue, Feb 5, 2019 at 6:09 PM Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov<mailto:shrirang.abhyankar at pnnl.gov>> wrote:
Hi Justin,
   Typically, the power grid distribution systems have a radial structure (unless it is an urban area) that leads to a, more or less, staircase type matrix. So a MatLoad() or VecLoad() would presumably just just splits the stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the phases for a bus to be split across processors, but I don?t think it would cause a big performance hit.
I?ve worked with OpenDSS recently and have created a C wrapper for it that you can use to query values from it. I can help out with your experimentations, if you need.

Shri
From: petsc-users <petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>> On Behalf Of Justin Chang via petsc-users
Sent: Tuesday, February 5, 2019 5:23 PM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers

Hi all,

So I *think* I know what the problem is. I am modeling a a distribution system where each bus can have anywhere from 1 to 3 phases. In the smallest example I had, all four buses had 3 phases (hence 12 dofs), and this was very easy to decompose - each MPI process should have a specific set of complex voltages on their respective nodes/buses. The larger test cases have lots of single phase buses attached, so a naive decomposition of the matrix isn't trivial. Again we're only extracting the Y-bus matrices out of an external software (OpenDSS to be specific) so we have no a priori information on how they organize their degrees-of-freedom - we let MatLoad() and VecLoad() do the decomposition. So it's possible that during this decomposition, a single bus' phases are being split across two or more MPI processes, which I assume would mess up the algorithmic performance of GAMG/ASM when more than one MPI process is needed.

Yeah DMNetwork will fix this and ensure that all dofs per vertex do not get split across different MPI procs, but for now we're just trying to see if we can get a simple proof-of-concept working. We're trying to come up with a toy distribution system where everything is three-phase (hence very straightforward to decompose even with MatLoad().

Justin

On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
I would stay away from eigen estimates in the solver (but give us the spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.

Applications that have lived on direct solvers can add sorts of crap like penalty terms.

sor seemed to work OK so I'd check the coarse grids in GAMG. Test with just two levels. That way you don't have to use sor on an internal MG grid, which can be bad and probably is here.


On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


> On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>
> Thanks everyone for your suggestions/feedback. So a few things:
>
> 1) When I examined larger distribution networks (~150k buses) my eigenvalue estimates from the chebyshev method get enormous indeed. See below:
>
> [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1, nnz/row (ave)=6, np=1
> [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with threshold 0., 6.01293 nnz ave. (N=320745)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00 min=4.639523e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
> [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with threshold 0., 7.32435 nnz ave. (N=44797)
> [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00 min=9.687506e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active pes
> [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with threshold 0., 22.5607 nnz ave. (N=13043)
> [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00 min=6.150856e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
> [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with threshold 0., 79.5142 nnz ave. (N=2256)
> [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00 min=6.780909e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
> [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with threshold 0., 206.379 nnz ave. (N=232)
> [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00 min=2.153627e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
>
> 2) I tried all the suggestions mentioned before: setting -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more iterations of richardson/sor.
>
> 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just solving LU on the whole system.
>
> 3b)The problem is that I can't get it to even speedup across 2 MPI processes for my 150k bus case (~300k dofs) - and I already checked that this problem could in theory be parallelized by setting the -ksp_max_it to something low and observing that the KSPSolve time decreases as MPI concurrency increases. The potential speedup is countered by the fact that the algorithmic convergence rate blows up when either more MPI processes are added or when I tune the block_size/overlap parameters.

   Yes, we just haven't gotten a scalable preconditioner yet for your matrix.

>
> Which leads me to one of two conclusions/questions:
>
> A) Is my 300 DOF problem still too small? Do I need to look at a problem with, say, 10 Million DOF or more to see that increasing the asm_block_size will have better performance (despite having more KSP iterations) than a single giant asm_block?

   No, if you had a scalable preconditioner you would see good speed up with 300k unknowns, using 10 million won't help.

>
> B) Or is there something even more sinister about my system of equations in complex form?

  There is a technical term for matrices for which both GAMG and ASM do very poorly on: nasty :-)

  Have you tried using parallel LU (for example ./configure --download-superlu_dist). Direct LU is the last refuge of the unpreconditionable. How large will your matrices get?

   Barry

>
> Thanks,
> Justin
>
> On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> The coarse grid is getting set to SOR rather than LU.
>
>    Matt
>
> On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> I tried these options:
>
> -ksp_view
> -ksp_monitor_true_residual
> -ksp_type gmres
> -ksp_max_it 50
> -pc_type gamg
> -mg_coarse_pc_type sor
> -mg_levels_1_ksp_type richardson
> -mg_levels_2_ksp_type richardson
> -mg_levels_3_ksp_type richardson
> -mg_levels_4_ksp_type richardson
> -mg_levels_1_pc_type sor
> -mg_levels_2_pc_type sor
> -mg_levels_3_pc_type sor
> -mg_levels_4_pc_type sor
>
> And still have a non-converging solution:
>
>   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
>   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
>   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
>   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
>   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
>   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
>   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
>   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
>   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
>  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
>  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
>  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
>  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
>  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
>  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
>  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
>  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
>  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
>  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
>  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
>  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
>  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
>  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
>  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
>  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
>  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
>  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
>  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
>  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
>  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
>  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
>  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
>  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
>  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
>  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
>  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
>  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
>  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
>  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
>  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
>  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
>  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
>  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
>  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
>  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
>  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
>  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
>  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
>  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
>  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=5 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.   0.   0.
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>         Complexity:    grid = 1.31821
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=17, cols=17
>         total: nonzeros=163, allocated nonzeros=163
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 12 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=100, cols=100
>         total: nonzeros=1240, allocated nonzeros=1240
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=537, cols=537
>         total: nonzeros=5291, allocated nonzeros=5291
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=1541, cols=1541
>         total: nonzeros=8039, allocated nonzeros=8039
>         total number of mallocs used during MatSetValues calls =0
>           not using I-node routines
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 1 MPI processes
>       type: richardson
>         damping factor=1.
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 1 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 1 MPI processes
>         type: seqaij
>         rows=8541, cols=8541
>         total: nonzeros=46299, allocated nonzeros=46299
>         total number of mallocs used during MatSetValues calls =0
>           using I-node routines: found 5464 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=8541, cols=8541
>     total: nonzeros=46299, allocated nonzeros=46299
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 5464 nodes, limit used is 5
>
> Am I doing this right? Did I miss anything?
>
>
>
> On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Hi Mark,
>
> 1) So with these options:
>
> -ksp_type gmres
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -ksp_converged_reason
> -pc_type bjacobi
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
>   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
>   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
>   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
>   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> Linear solve converged due to CONVERGED_RTOL iterations 5
>
> 2) With richardson/sor:
>
> Okay, its looks like Richardson/SOR solves this just fine. You can use this as the smoother for GAMG instead
> of Cheby/Jacobi, and probably see better results on the larger problems.
>
>   Matt
>
> -ksp_type richardson
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type sor
>
> This is what I get:
>
>   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
>   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
>   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
>   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
>   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
>   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
>   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
>   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
>   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
>  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
>  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
>  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
>  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
>  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
>  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
>  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
>  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
>  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
>  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
>  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
>  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
>  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
>  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
>  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
>  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
>  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
>  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
>  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
>  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
>  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
>  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
>  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
>  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
>  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
>  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
>  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
>  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
>  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
>  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
>  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> Linear solve converged due to CONVERGED_RTOL iterations 40
>
> 3) And lastly with chebyshev/jacobi:
>
> -ksp_type chebyshev
> -ksp_rtol 1e-15
> -ksp_monitor_true_residual
> -pc_type jacobi
>
>   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
>   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
>   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
>   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
>   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
>   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
>   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
>   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
>   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
>   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
>  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
>  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
>  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
>  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
>  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
>  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
>  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
>  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
>  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
>  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
>  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
>  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
>  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
>  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
>  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
>  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
>  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
>  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
>  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
>  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
>  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
>  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
>  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
>  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
>  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
>  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
>  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
>  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
>  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
>  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
>  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
>  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
>  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
>  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
>  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
>  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
>  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
>  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
>  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
>  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
>  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
>  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
>  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
>  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
>  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
>  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
>  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
>  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
>  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
>  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
>  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
>  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
>  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
>  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
>  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
>  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
>  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> Linear solve converged due to CONVERGED_RTOL iterations 67
>
> Looks like neither of these ksp/pc combos are good? I also tried -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the GAMG info I was able to grep from the out-of-box params:
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00 min=2.394056e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 5.21674 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00 min=6.783380e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 9.85289 nnz ave. (N=537)
> [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00 min=5.974776e-02 PC=jacobi
> [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 12.4 nnz ave. (N=100)
> [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00 min=4.842076e-01 PC=jacobi
> [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
>
> And the one with pc_gamg_agg_nsmooths 0
>
> [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with threshold 0., 3.07398 nnz ave. (N=1541)
> [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.02211 nnz ave. (N=814)
> [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.00434 nnz ave. (N=461)
> [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=290)
> [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=197)
> [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.98425 nnz ave. (N=127)
> [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.97561 nnz ave. (N=82)
> [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.9697 nnz ave. (N=66)
> [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
>
>
> Thanks,
> Justin
>
> On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
>
>
> Both GAMG and ILU are nice and dandy for this,
>
> I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and converge it way down, say rtol = 1.e-12. See which one is better in the early iteration and pick it. It would be nice to check that it solves the problem ...
>
> The residual drops about 5 orders in the first iteration and then flatlines. That is very bad. Check that the smoother can actually solve the problem.
>
> but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG.
>
> First try -pc_gamg_nsmooths 0
>
> You can run with -info and grep on GAMG to see some diagnostic output. Eigen estimates are fragile in practice and with this parameter and SOR there are no eigen estimates needed. The max eigen values for all levels should be between say 2 (or less) and say 3-4. Much higher is a sign of a problem.
>
> Attached is the ksp monitor/view output for gamg on the unsuccessful solve
>
> I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
>
> Thanks
>
> Justin
>
> PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
>
> There is a write up in the manual that is tutorial like.
>
>
> On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com<mailto:jychang48 at gmail.com>> wrote:
> Here's IMHO the simplest explanation of the equations I'm trying to solve:
>
> http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
>
> Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
>
> Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> small examples.
>
>   Thanks,
>
>     Matt
>
> On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
> On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> Hi all,
>
> I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers.
>
> I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
>
> So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
>
> 1) We really need to see the (simplified) equations
>
> 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
>
>  Thanks,
>
>     Matt
>
> Justin
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190211/58dd53d6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 25884 bytes
Desc: image001.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190211/58dd53d6/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 19966 bytes
Desc: image002.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190211/58dd53d6/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 15018 bytes
Desc: image003.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190211/58dd53d6/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 18636 bytes
Desc: image004.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190211/58dd53d6/attachment-0007.png>

From ztdepyahoo at gmail.com  Mon Feb 11 18:59:49 2019
From: ztdepyahoo at gmail.com (ztdepyahoo)
Date: Tue, 12 Feb 2019 08:59:49 +0800
Subject: [petsc-users] About the value of the PETSC_SMALL
Message-ID: <83B620F5-ADDB-47E3-9D2B-BD4617D70283@gmail.com>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/4f81a65c/attachment.html>

From jed at jedbrown.org  Mon Feb 11 21:03:38 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 Feb 2019 20:03:38 -0700
Subject: [petsc-users] About the value of the PETSC_SMALL
In-Reply-To: <83B620F5-ADDB-47E3-9D2B-BD4617D70283@gmail.com>
References: <83B620F5-ADDB-47E3-9D2B-BD4617D70283@gmail.com>
Message-ID: <874l99emid.fsf@jedbrown.org>

You're probably looking for PETSC_MACHINE_EPSILON.

ztdepyahoo via petsc-users <petsc-users at mcs.anl.gov> writes:

> Dear sir:
>     I output the value  of the "PETSC_SMALL", it is 1E-10.  But i think it
> should be more smaller than this for double float number.
> Regards

From bsmith at mcs.anl.gov  Mon Feb 11 21:16:10 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 12 Feb 2019 03:16:10 +0000
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <DM6PR09MB3244E1F15AC69972794BCD95E7640@DM6PR09MB3244.namprd09.prod.outlook.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
	<DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>
	<DM6PR09MB3244E1F15AC69972794BCD95E7640@DM6PR09MB3244.namprd09.prod.outlook.com>
Message-ID: <08452B05-1269-4B9B-9018-89D2FE41BF24@anl.gov>


   Given the nonzero structure of the matrix I'd be surprised if either GAMG/boomerAMG or ASM was particularly efficacious. I don't have any recommendations for iterative methods.


   Barry


> On Feb 11, 2019, at 8:41 AM, Abhyankar, Shrirang G via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
>  
>  
> From: Justin Chang <jychang48 at gmail.com> 
> Sent: Friday, February 8, 2019 4:57 PM
> To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
> Cc: Mark Adams <mfadams at lbl.gov>; PETSc users list <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers
>  
> So I used -mat_view draw -draw_pause -1 on my medium sized matrix and got this output:
>  
> <image001.png>
>  
> So it seems there are lots of off-diagonal terms, and that a decomposition of the problem via matload would give a terrible unbalanced problem.
>  
> Given the initial A and b Mat/Vec, I experimented with MatPartioning and inserted the following lines into my code:
>  
> Mat               Apart;
> Vec               bpart;
> MatPartitioning   part;
> IS                is,isrows;
> ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part);CHKERRQ(ierr);
> ierr = MatPartitioningSetAdjacency(part, A);CHKERRQ(ierr);
> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
> ierr = MatPartitioningApply(part, &is);CHKERRQ(ierr);
> ierr = ISBuildTwoSided(is,NULL,&isrows);CHKERRQ(ierr);
> ierr = MatCreateSubMatrix(A, isrows,isrows, MAT_INITIAL_MATRIX, &Apart);CHKERRQ(ierr);
> ierr = MatSetOptionsPrefix(Apart, "part_");CHKERRQ(ierr);
> ierr = MatSetFromOptions(Apart);CHKERRQ(ierr);
> ierr = VecGetSubVector(b,isrows,&bpart);CHKERRQ(ierr);
>  
> /* Set Apart and bpart in the KSPSolve */
> ...
>  
> And here are the mat_draw figures from 2 and 4 MPI processes respectively:
>  
> <image002.png><image003.png>
>  
> Is this "right"? It just feels like I'm just duplicating the nnz structure among all the MPI processes. And it didn't really improve the performance of ASM. 
>  
> This looks kind of right. You have the big fat diagonal blocks represent separate meshed networks, and the sparse off-diagonal blocks that show very less connectivity between the networks. This is typical of power grid where the connectivity between different areas is sparse. However, the blocks being identical is puzzling. I would expect the blocks to be of different sizes. Was this network created by duplicating smaller networks?
> I was hoping ASM would be faster on the partitioned matrix rather than the original one. What options are you using for sub_pc? -sub_pc_type lu -sub_pc_factor_mat_ordering_type amd?
>  
> Also, this nnz structure appears to depend on the distribution system we cook up in OpenDSS, for example the largest matrix we have thus far looks like this:
> <image004.png>
> Yes, the matrix structure depends on the network structure. This to me looks like a star-shaped network where you have a tightly interconnected network (bottom dense block) and smallish radial distribution networks emanating from the nodes. That?s just one guess. It could be some other type as well.
>  
>  
> Is there a better alternative to the partitioning I implemented above? Or did I do something catastrophically wrong
>  
> Thanks,
> Justin
>  
> Side node - Yes I tried KLU, but it needed several KSP iterations to obtain the solution. Here's the ksp monitor/view outputs:
>  
> 0 KSP preconditioned resid norm 1.705434112839e+06 true resid norm 2.242813827253e+12 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.122965789284e+05 true resid norm 3.057749589444e+11 ||r(i)||/||b|| 1.363354172463e-01
>   2 KSP preconditioned resid norm 1.962518730076e+04 true resid norm 2.510054932552e+10 ||r(i)||/||b|| 1.119154386357e-02
>   3 KSP preconditioned resid norm 3.094963519133e+03 true resid norm 6.489763653495e+09 ||r(i)||/||b|| 2.893581078660e-03
>   4 KSP preconditioned resid norm 1.755871992454e+03 true resid norm 2.315676037474e+09 ||r(i)||/||b|| 1.032486963178e-03
>   5 KSP preconditioned resid norm 1.348939340771e+03 true resid norm 1.864929933344e+09 ||r(i)||/||b|| 8.315134812722e-04
>   6 KSP preconditioned resid norm 5.532203694243e+02 true resid norm 9.985525631209e+08 ||r(i)||/||b|| 4.452231170449e-04
>   7 KSP preconditioned resid norm 3.636087020506e+02 true resid norm 5.712899201028e+08 ||r(i)||/||b|| 2.547201703329e-04
>   8 KSP preconditioned resid norm 2.926812321412e+02 true resid norm 3.627282296417e+08 ||r(i)||/||b|| 1.617290856843e-04
>   9 KSP preconditioned resid norm 1.629184033135e+02 true resid norm 1.048838851435e+08 ||r(i)||/||b|| 4.676441881580e-05
>  10 KSP preconditioned resid norm 8.297821067807e+01 true resid norm 3.423640694920e+07 ||r(i)||/||b|| 1.526493484800e-05
>  11 KSP preconditioned resid norm 2.997246200648e+01 true resid norm 9.880250538293e+06 ||r(i)||/||b|| 4.405292324417e-06
>  12 KSP preconditioned resid norm 2.156940809471e+01 true resid norm 3.521518932572e+06 ||r(i)||/||b|| 1.570134306192e-06
>  13 KSP preconditioned resid norm 1.211823308437e+01 true resid norm 1.130205443318e+06 ||r(i)||/||b|| 5.039229870908e-07
> Linear solve converged due to CONVERGED_RTOL iterations 13
> KSP Object: 1 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=50, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: lu
>     out-of-place factorization
>     tolerance for zero pivot 2.22045e-14
>     matrix ordering: amd
>     factor fill ratio given 0., needed 0.
>       Factored matrix follows:
>         Mat Object: 1 MPI processes
>           type: klu
>           rows=320745, cols=320745
>           package used to perform factorization: klu
>           total: nonzeros=0, allocated nonzeros=0
>           total number of mallocs used during MatSetValues calls =0
>             KLU stats:
>               Number of diagonal blocks: 1
>               Total nonzeros=2426908
>             KLU runtime parameters:
>               Partial pivoting tolerance: 0.001
>               BTF preordering enabled: 1
>               Ordering: AMD (not using the PETSc ordering)
>               Matrix row scaling: NONE
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=320745, cols=320745
>     total: nonzeros=1928617, allocated nonzeros=1928617
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 186383 nodes, limit used is 5
>  
> It was slower than the PETSc LU solver which doesn't seem to make sense to me. The lines:
>  
> Hmm?that is strange. KLU is a direct solver so it should be as fast as PETSc?s direct solver. Can you send me the matrix and the code (offline). I?d like to test.
>  
> Shri
>  total: nonzeros=0, allocated nonzeros=0
>  total number of mallocs used during MatSetValues calls =0
>  
> seem to indicate that KLU needs to be customized to get the "high performance" that the circuits community claim to have?
>  
> On Thu, Feb 7, 2019 at 8:03 AM Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov> wrote:
> Hi Justin, 
>     The serial solve time for this matrix of size 320K is about 0.3 secs, which I think is pretty fast. As such, I am not surprised you are not getting any speed up in parallel with asm. Perhaps, you need a bigger matrix?
>  
> It would be interesting to see a spy plot of the matrix on different processors. If there are a lot of off-diagonal elements, then asm may not work that well. You may need to use a partitioner in that case.
>  
> As a side note, folks in the circuits community consider KLU as one of the fastest serial solvers for such kind of problems. You can try that out and see if it improves serial performance even further.
>  
> --download-suitesparse
>  
> -pc_factor_mat_solver_package klu -pc_factor_mat_ordering_type amd
>  
> Shri
>  
> From: Justin Chang <jychang48 at gmail.com> 
> Sent: Wednesday, February 6, 2019 2:56 PM
> To: Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov>
> Cc: Mark Adams <mfadams at lbl.gov>; PETSc users list <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers
>  
> Hi Shri,
>  
> Yeah so here at NREL I'm working with a group of folks (specifically Dheepak Krishnamurthy, who apparently knows you back when y'all were both at Argonne) and asked them to cook up a urban-suburban distribution system from OpenDSS (he wrote a Julia/Python wrapper for OpenDSS). I used the following options:
> 
> -ksp_type gmres
> -pc_type asm 
> -pc_asm_overlap 3
> -sub_pc_type lu
> -sub_pc_factor_mat_ordering_type amd # using rcm also produced similar results, all of which better than the default
> -ksp_monitor_true_residual
> -ksp_view
> -log_view
>  
> And here are the long overdue outputs for up to 16 MPI processes on our Eagle HPC system (with Skylake server nodes).
>  
> Basically, the algorithmic convergence degrades when more MPI procs are used, and while I could ameliorate the KSP count by increasing the overlap, it slowdowns the overall computation which I assume is from the communication latencies. Also note that the KSPSolve time also slowed down because of the ksp_monitor
>  
> Is this a "big performance hit"? I don't know, maybe I need to first start off with a fully 3-phase system and see if similar solver characteristics are found?
>  
> Thanks,
> Justin
>  
>  
>  
>  
>  
> On Tue, Feb 5, 2019 at 6:09 PM Abhyankar, Shrirang G <shrirang.abhyankar at pnnl.gov> wrote:
> Hi Justin,
>    Typically, the power grid distribution systems have a radial structure (unless it is an urban area) that leads to a, more or less, staircase type matrix. So a MatLoad() or VecLoad() would presumably just just splits the stairs, akin to a 1-D PDE. However, as you pointed out, it may cause the phases for a bus to be split across processors, but I don?t think it would cause a big performance hit.
> I?ve worked with OpenDSS recently and have created a C wrapper for it that you can use to query values from it. I can help out with your experimentations, if you need.
>  
> Shri 
> From: petsc-users <petsc-users-bounces at mcs.anl.gov> On Behalf Of Justin Chang via petsc-users
> Sent: Tuesday, February 5, 2019 5:23 PM
> To: Mark Adams <mfadams at lbl.gov>
> Cc: petsc-users <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Preconditioning systems of equations with complex numbers
>  
> Hi all,
>  
> So I *think* I know what the problem is. I am modeling a a distribution system where each bus can have anywhere from 1 to 3 phases. In the smallest example I had, all four buses had 3 phases (hence 12 dofs), and this was very easy to decompose - each MPI process should have a specific set of complex voltages on their respective nodes/buses. The larger test cases have lots of single phase buses attached, so a naive decomposition of the matrix isn't trivial. Again we're only extracting the Y-bus matrices out of an external software (OpenDSS to be specific) so we have no a priori information on how they organize their degrees-of-freedom - we let MatLoad() and VecLoad() do the decomposition. So it's possible that during this decomposition, a single bus' phases are being split across two or more MPI processes, which I assume would mess up the algorithmic performance of GAMG/ASM when more than one MPI process is needed.
>  
> Yeah DMNetwork will fix this and ensure that all dofs per vertex do not get split across different MPI procs, but for now we're just trying to see if we can get a simple proof-of-concept working. We're trying to come up with a toy distribution system where everything is three-phase (hence very straightforward to decompose even with MatLoad().
>  
> Justin
>  
> On Tue, Feb 5, 2019 at 9:26 AM Mark Adams <mfadams at lbl.gov> wrote:
> I would stay away from eigen estimates in the solver (but give us the spectra to look at), so set -pc_gamg_agg_nsmooths 0 and use sor.
>  
> Applications that have lived on direct solvers can add sorts of crap like penalty terms.
>  
> sor seemed to work OK so I'd check the coarse grids in GAMG. Test with just two levels. That way you don't have to use sor on an internal MG grid, which can be bad and probably is here.
>  
>  
> On Mon, Feb 4, 2019 at 10:59 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> > On Feb 4, 2019, at 1:56 PM, Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Thanks everyone for your suggestions/feedback. So a few things:
> > 
> > 1) When I examined larger distribution networks (~150k buses) my eigenvalue estimates from the chebyshev method get enormous indeed. See below:
> > 
> > [0] PCSetUp_GAMG(): level 0) N=320745, n data rows=1, n data cols=1, nnz/row (ave)=6, np=1
> > [0] PCGAMGFilterGraph():       98.5638% nnz after filtering, with threshold 0., 6.01293 nnz ave. (N=320745)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 44797 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.403335e+00 min=4.639523e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=44797, n data cols=1, nnz/row (ave)=7, 1 active pes
> > [0] PCGAMGFilterGraph():       99.9753% nnz after filtering, with threshold 0., 7.32435 nnz ave. (N=44797)
> > [0] PCGAMGProlongator_AGG(): New grid 13043 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=8.173298e+00 min=9.687506e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=13043, n data cols=1, nnz/row (ave)=22, 1 active pes
> > [0] PCGAMGFilterGraph():       99.684% nnz after filtering, with threshold 0., 22.5607 nnz ave. (N=13043)
> > [0] PCGAMGProlongator_AGG(): New grid 2256 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=5.696594e+00 min=6.150856e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=2256, n data cols=1, nnz/row (ave)=79, 1 active pes
> > [0] PCGAMGFilterGraph():       93.859% nnz after filtering, with threshold 0., 79.5142 nnz ave. (N=2256)
> > [0] PCGAMGProlongator_AGG(): New grid 232 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.454120e+00 min=6.780909e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=232, n data cols=1, nnz/row (ave)=206, 1 active pes
> > [0] PCGAMGFilterGraph():       99.1729% nnz after filtering, with threshold 0., 206.379 nnz ave. (N=232)
> > [0] PCGAMGProlongator_AGG(): New grid 9 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.443612e+00 min=2.153627e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 5) N=9, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 6 levels, grid complexity = 1.44058
> > 
> > 2) I tried all the suggestions mentioned before: setting -pc_gamg_agg_nsmooths 0  -pc_gamg_square_graph 10 did not improve my convergence. Neither did explicitly setting -mg_coarse_pc_type lu or more iterations of richardson/sor.
> > 
> > 3a) -pc_type asm works only if I set -sub_pc_type lu. Basically I'm just solving LU on the whole system.
> > 
> > 3b)The problem is that I can't get it to even speedup across 2 MPI processes for my 150k bus case (~300k dofs) - and I already checked that this problem could in theory be parallelized by setting the -ksp_max_it to something low and observing that the KSPSolve time decreases as MPI concurrency increases. The potential speedup is countered by the fact that the algorithmic convergence rate blows up when either more MPI processes are added or when I tune the block_size/overlap parameters. 
> 
>    Yes, we just haven't gotten a scalable preconditioner yet for your matrix.
> 
> > 
> > Which leads me to one of two conclusions/questions:
> > 
> > A) Is my 300 DOF problem still too small? Do I need to look at a problem with, say, 10 Million DOF or more to see that increasing the asm_block_size will have better performance (despite having more KSP iterations) than a single giant asm_block?
> 
>    No, if you had a scalable preconditioner you would see good speed up with 300k unknowns, using 10 million won't help.
> 
> > 
> > B) Or is there something even more sinister about my system of equations in complex form?
> 
>   There is a technical term for matrices for which both GAMG and ASM do very poorly on: nasty :-)
> 
>   Have you tried using parallel LU (for example ./configure --download-superlu_dist). Direct LU is the last refuge of the unpreconditionable. How large will your matrices get? 
> 
>    Barry
> 
> > 
> > Thanks,
> > Justin
> > 
> > On Sat, Feb 2, 2019 at 1:41 PM Matthew Knepley <knepley at gmail.com> wrote:
> > The coarse grid is getting set to SOR rather than LU.
> > 
> >    Matt
> > 
> > On Fri, Feb 1, 2019 at 3:54 PM Justin Chang <jychang48 at gmail.com> wrote:
> > I tried these options:
> > 
> > -ksp_view
> > -ksp_monitor_true_residual
> > -ksp_type gmres
> > -ksp_max_it 50
> > -pc_type gamg
> > -mg_coarse_pc_type sor
> > -mg_levels_1_ksp_type richardson
> > -mg_levels_2_ksp_type richardson
> > -mg_levels_3_ksp_type richardson
> > -mg_levels_4_ksp_type richardson
> > -mg_levels_1_pc_type sor
> > -mg_levels_2_pc_type sor
> > -mg_levels_3_pc_type sor
> > -mg_levels_4_pc_type sor
> > 
> > And still have a non-converging solution:
> > 
> >   0 KSP preconditioned resid norm 1.573625743885e+05 true resid norm 1.207500000000e+08 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 7.150979785488e+00 true resid norm 1.218223835173e+04 ||r(i)||/||b|| 1.008881022917e-04
> >   2 KSP preconditioned resid norm 5.627140225062e+00 true resid norm 1.209346465397e+04 ||r(i)||/||b|| 1.001529163889e-04
> >   3 KSP preconditioned resid norm 5.077324098998e+00 true resid norm 1.211293532189e+04 ||r(i)||/||b|| 1.003141641564e-04
> >   4 KSP preconditioned resid norm 4.324840589068e+00 true resid norm 1.213758867766e+04 ||r(i)||/||b|| 1.005183327342e-04
> >   5 KSP preconditioned resid norm 4.107764194462e+00 true resid norm 1.212180975158e+04 ||r(i)||/||b|| 1.003876583981e-04
> >   6 KSP preconditioned resid norm 4.033152911227e+00 true resid norm 1.216543340208e+04 ||r(i)||/||b|| 1.007489308660e-04
> >   7 KSP preconditioned resid norm 3.944875788743e+00 true resid norm 1.220023550040e+04 ||r(i)||/||b|| 1.010371470012e-04
> >   8 KSP preconditioned resid norm 3.926562233824e+00 true resid norm 1.220662039610e+04 ||r(i)||/||b|| 1.010900239842e-04
> >   9 KSP preconditioned resid norm 3.912450246357e+00 true resid norm 1.219755005917e+04 ||r(i)||/||b|| 1.010149073223e-04
> >  10 KSP preconditioned resid norm 3.838058250285e+00 true resid norm 1.218625848551e+04 ||r(i)||/||b|| 1.009213953251e-04
> >  11 KSP preconditioned resid norm 3.779259857769e+00 true resid norm 1.219659071592e+04 ||r(i)||/||b|| 1.010069624506e-04
> >  12 KSP preconditioned resid norm 3.694467405997e+00 true resid norm 1.223911592358e+04 ||r(i)||/||b|| 1.013591380834e-04
> >  13 KSP preconditioned resid norm 3.693468937696e+00 true resid norm 1.223724162745e+04 ||r(i)||/||b|| 1.013436159623e-04
> >  14 KSP preconditioned resid norm 3.616103662529e+00 true resid norm 1.221242236516e+04 ||r(i)||/||b|| 1.011380734175e-04
> >  15 KSP preconditioned resid norm 3.604541982832e+00 true resid norm 1.220338362332e+04 ||r(i)||/||b|| 1.010632184126e-04
> >  16 KSP preconditioned resid norm 3.599588304412e+00 true resid norm 1.219870618529e+04 ||r(i)||/||b|| 1.010244818657e-04
> >  17 KSP preconditioned resid norm 3.581088429735e+00 true resid norm 1.219098863716e+04 ||r(i)||/||b|| 1.009605684237e-04
> >  18 KSP preconditioned resid norm 3.577183857191e+00 true resid norm 1.219422257290e+04 ||r(i)||/||b|| 1.009873505002e-04
> >  19 KSP preconditioned resid norm 3.576075374018e+00 true resid norm 1.219687565514e+04 ||r(i)||/||b|| 1.010093221958e-04
> >  20 KSP preconditioned resid norm 3.574659675290e+00 true resid norm 1.219494681037e+04 ||r(i)||/||b|| 1.009933483260e-04
> >  21 KSP preconditioned resid norm 3.555091405270e+00 true resid norm 1.218714149002e+04 ||r(i)||/||b|| 1.009287079919e-04
> >  22 KSP preconditioned resid norm 3.553181473875e+00 true resid norm 1.218106845443e+04 ||r(i)||/||b|| 1.008784137012e-04
> >  23 KSP preconditioned resid norm 3.552613734076e+00 true resid norm 1.217794060488e+04 ||r(i)||/||b|| 1.008525101853e-04
> >  24 KSP preconditioned resid norm 3.551632777626e+00 true resid norm 1.218032078237e+04 ||r(i)||/||b|| 1.008722218001e-04
> >  25 KSP preconditioned resid norm 3.545808514701e+00 true resid norm 1.219292407751e+04 ||r(i)||/||b|| 1.009765969151e-04
> >  26 KSP preconditioned resid norm 3.528978940908e+00 true resid norm 1.219073670770e+04 ||r(i)||/||b|| 1.009584820513e-04
> >  27 KSP preconditioned resid norm 3.527789136030e+00 true resid norm 1.218958906209e+04 ||r(i)||/||b|| 1.009489777399e-04
> >  28 KSP preconditioned resid norm 3.525383863095e+00 true resid norm 1.218344768375e+04 ||r(i)||/||b|| 1.008981174637e-04
> >  29 KSP preconditioned resid norm 3.521750505784e+00 true resid norm 1.218775781359e+04 ||r(i)||/||b|| 1.009338121208e-04
> >  30 KSP preconditioned resid norm 3.521656008348e+00 true resid norm 1.218930075437e+04 ||r(i)||/||b|| 1.009465900983e-04
> >  31 KSP preconditioned resid norm 3.521655969692e+00 true resid norm 1.218928102429e+04 ||r(i)||/||b|| 1.009464267021e-04
> >  32 KSP preconditioned resid norm 3.521654896823e+00 true resid norm 1.218929084207e+04 ||r(i)||/||b|| 1.009465080088e-04
> >  33 KSP preconditioned resid norm 3.521654041929e+00 true resid norm 1.218930804845e+04 ||r(i)||/||b|| 1.009466505047e-04
> >  34 KSP preconditioned resid norm 3.521652043655e+00 true resid norm 1.218914044919e+04 ||r(i)||/||b|| 1.009452625192e-04
> >  35 KSP preconditioned resid norm 3.521602884006e+00 true resid norm 1.218810333491e+04 ||r(i)||/||b|| 1.009366735810e-04
> >  36 KSP preconditioned resid norm 3.521535454292e+00 true resid norm 1.218820358646e+04 ||r(i)||/||b|| 1.009375038215e-04
> >  37 KSP preconditioned resid norm 3.521433576778e+00 true resid norm 1.218859696943e+04 ||r(i)||/||b|| 1.009407616516e-04
> >  38 KSP preconditioned resid norm 3.521349747881e+00 true resid norm 1.218888798454e+04 ||r(i)||/||b|| 1.009431717146e-04
> >  39 KSP preconditioned resid norm 3.521212709133e+00 true resid norm 1.218882278104e+04 ||r(i)||/||b|| 1.009426317270e-04
> >  40 KSP preconditioned resid norm 3.521168785360e+00 true resid norm 1.218866389996e+04 ||r(i)||/||b|| 1.009413159416e-04
> >  41 KSP preconditioned resid norm 3.521164077366e+00 true resid norm 1.218868324624e+04 ||r(i)||/||b|| 1.009414761593e-04
> >  42 KSP preconditioned resid norm 3.521101506147e+00 true resid norm 1.218773304385e+04 ||r(i)||/||b|| 1.009336069884e-04
> >  43 KSP preconditioned resid norm 3.521013554688e+00 true resid norm 1.218675768694e+04 ||r(i)||/||b|| 1.009255294984e-04
> >  44 KSP preconditioned resid norm 3.520820039115e+00 true resid norm 1.218829726209e+04 ||r(i)||/||b|| 1.009382796032e-04
> >  45 KSP preconditioned resid norm 3.520763991575e+00 true resid norm 1.218776399191e+04 ||r(i)||/||b|| 1.009338632870e-04
> >  46 KSP preconditioned resid norm 3.520501770928e+00 true resid norm 1.218624167053e+04 ||r(i)||/||b|| 1.009212560706e-04
> >  47 KSP preconditioned resid norm 3.519005707047e+00 true resid norm 1.219233855078e+04 ||r(i)||/||b|| 1.009717478325e-04
> >  48 KSP preconditioned resid norm 3.518379807717e+00 true resid norm 1.218961932675e+04 ||r(i)||/||b|| 1.009492283788e-04
> >  49 KSP preconditioned resid norm 3.517809415824e+00 true resid norm 1.218777984143e+04 ||r(i)||/||b|| 1.009339945460e-04
> >  50 KSP preconditioned resid norm 3.517617854442e+00 true resid norm 1.219011629981e+04 ||r(i)||/||b|| 1.009533440978e-04
> > KSP Object: 1 MPI processes
> >   type: gmres
> >     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> >     happy breakdown tolerance 1e-30
> >   maximum iterations=50, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: gamg
> >     type is MULTIPLICATIVE, levels=5 cycles=v
> >       Cycles per PCApply=1
> >       Using externally compute Galerkin coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values in graph on each level =   0.   0.   0.  
> >         Threshold scaling factor for each level not specified = 1.
> >         AGG specific options
> >           Symmetric graph false
> >           Number of levels to square graph 1
> >           Number smoothing steps 1
> >         Complexity:    grid = 1.31821
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object: (mg_coarse_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_coarse_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=17, cols=17
> >         total: nonzeros=163, allocated nonzeros=163
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 12 nodes, limit used is 5
> >   Down solver (pre-smoother) on level 1 -------------------------------
> >     KSP Object: (mg_levels_1_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_1_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=100, cols=100
> >         total: nonzeros=1240, allocated nonzeros=1240
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 2 -------------------------------
> >     KSP Object: (mg_levels_2_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_2_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=537, cols=537
> >         total: nonzeros=5291, allocated nonzeros=5291
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 3 -------------------------------
> >     KSP Object: (mg_levels_3_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_3_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=1541, cols=1541
> >         total: nonzeros=8039, allocated nonzeros=8039
> >         total number of mallocs used during MatSetValues calls =0
> >           not using I-node routines
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   Down solver (pre-smoother) on level 4 -------------------------------
> >     KSP Object: (mg_levels_4_) 1 MPI processes
> >       type: richardson
> >         damping factor=1.
> >       maximum iterations=2, nonzero initial guess
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using NONE norm type for convergence test
> >     PC Object: (mg_levels_4_) 1 MPI processes
> >       type: sor
> >         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
> >       linear system matrix = precond matrix:
> >       Mat Object: 1 MPI processes
> >         type: seqaij
> >         rows=8541, cols=8541
> >         total: nonzeros=46299, allocated nonzeros=46299
> >         total number of mallocs used during MatSetValues calls =0
> >           using I-node routines: found 5464 nodes, limit used is 5
> >   Up solver (post-smoother) same as down solver (pre-smoother)
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> >     type: seqaij
> >     rows=8541, cols=8541
> >     total: nonzeros=46299, allocated nonzeros=46299
> >     total number of mallocs used during MatSetValues calls =0
> >       using I-node routines: found 5464 nodes, limit used is 5
> > 
> > Am I doing this right? Did I miss anything?
> > 
> > 
> > 
> > On Fri, Feb 1, 2019 at 1:22 PM Matthew Knepley <knepley at gmail.com> wrote:
> > On Fri, Feb 1, 2019 at 3:05 PM Justin Chang <jychang48 at gmail.com> wrote:
> > Hi Mark,
> > 
> > 1) So with these options:
> > 
> > -ksp_type gmres
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -ksp_converged_reason
> > -pc_type bjacobi
> > 
> > This is what I get:
> > 
> >   0 KSP preconditioned resid norm 1.900749341028e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.363172190058e+03 true resid norm 8.144804150077e+05 ||r(i)||/||b|| 5.078285384488e-02
> >   2 KSP preconditioned resid norm 1.744431536443e+02 true resid norm 2.668646207476e+04 ||r(i)||/||b|| 1.663900909350e-03
> >   3 KSP preconditioned resid norm 2.798960448093e+01 true resid norm 2.142171349084e+03 ||r(i)||/||b|| 1.335643835343e-04
> >   4 KSP preconditioned resid norm 2.938319245576e+00 true resid norm 1.457432872748e+03 ||r(i)||/||b|| 9.087093956061e-05
> >   5 KSP preconditioned resid norm 1.539484356450e-12 true resid norm 6.667274739274e-09 ||r(i)||/||b|| 4.157045797412e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 5
> > 
> > 2) With richardson/sor:
> > 
> > Okay, its looks like Richardson/SOR solves this just fine. You can use this as the smoother for GAMG instead
> > of Cheby/Jacobi, and probably see better results on the larger problems.
> > 
> >   Matt
> >  
> > -ksp_type richardson
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type sor
> > 
> > This is what I get:
> > 
> >   0 KSP preconditioned resid norm 1.772935756018e+04 true resid norm 1.603849239146e+07 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.206881305953e+03 true resid norm 4.507533687463e+03 ||r(i)||/||b|| 2.810447252426e-04
> >   2 KSP preconditioned resid norm 4.166906741810e+02 true resid norm 1.221098715634e+03 ||r(i)||/||b|| 7.613550487354e-05
> >   3 KSP preconditioned resid norm 1.540698682668e+02 true resid norm 4.241154731706e+02 ||r(i)||/||b|| 2.644359973612e-05
> >   4 KSP preconditioned resid norm 5.904921520051e+01 true resid norm 1.587778309916e+02 ||r(i)||/||b|| 9.899797756316e-06
> >   5 KSP preconditioned resid norm 2.327938633860e+01 true resid norm 6.161476099609e+01 ||r(i)||/||b|| 3.841680345773e-06
> >   6 KSP preconditioned resid norm 9.409043410169e+00 true resid norm 2.458600025207e+01 ||r(i)||/||b|| 1.532937114786e-06
> >   7 KSP preconditioned resid norm 3.888365194933e+00 true resid norm 1.005868527582e+01 ||r(i)||/||b|| 6.271590265661e-07
> >   8 KSP preconditioned resid norm 1.638018293396e+00 true resid norm 4.207888403975e+00 ||r(i)||/||b|| 2.623618418285e-07
> >   9 KSP preconditioned resid norm 7.010639340830e-01 true resid norm 1.793827818698e+00 ||r(i)||/||b|| 1.118451644279e-07
> >  10 KSP preconditioned resid norm 3.038491129050e-01 true resid norm 7.763187747256e-01 ||r(i)||/||b|| 4.840347557474e-08
> >  11 KSP preconditioned resid norm 1.329641892383e-01 true resid norm 3.398137553975e-01 ||r(i)||/||b|| 2.118738763617e-08
> >  12 KSP preconditioned resid norm 5.860142364318e-02 true resid norm 1.499670452314e-01 ||r(i)||/||b|| 9.350445264501e-09
> >  13 KSP preconditioned resid norm 2.596075957908e-02 true resid norm 6.655772419557e-02 ||r(i)||/||b|| 4.149874101073e-09
> >  14 KSP preconditioned resid norm 1.154254160823e-02 true resid norm 2.964959279341e-02 ||r(i)||/||b|| 1.848652109546e-09
> >  15 KSP preconditioned resid norm 5.144785556436e-03 true resid norm 1.323918323132e-02 ||r(i)||/||b|| 8.254630739714e-10
> >  16 KSP preconditioned resid norm 2.296969429446e-03 true resid norm 5.919889645162e-03 ||r(i)||/||b|| 3.691051191517e-10
> >  17 KSP preconditioned resid norm 1.026615599876e-03 true resid norm 2.649106197813e-03 ||r(i)||/||b|| 1.651717713333e-10
> >  18 KSP preconditioned resid norm 4.591391433184e-04 true resid norm 1.185874433977e-03 ||r(i)||/||b|| 7.393927091354e-11
> >  19 KSP preconditioned resid norm 2.054186999728e-04 true resid norm 5.309080126683e-04 ||r(i)||/||b|| 3.310211456976e-11
> >  20 KSP preconditioned resid norm 9.192021190159e-05 true resid norm 2.376701584665e-04 ||r(i)||/||b|| 1.481873437138e-11
> >  21 KSP preconditioned resid norm 4.113417679473e-05 true resid norm 1.063822174067e-04 ||r(i)||/||b|| 6.632931251279e-12
> >  22 KSP preconditioned resid norm 1.840693141405e-05 true resid norm 4.760872579965e-05 ||r(i)||/||b|| 2.968404051805e-12
> >  23 KSP preconditioned resid norm 8.236207862555e-06 true resid norm 2.130217091325e-05 ||r(i)||/||b|| 1.328190355634e-12
> >  24 KSP preconditioned resid norm 3.684941963736e-06 true resid norm 9.529827966741e-06 ||r(i)||/||b|| 5.941847733654e-13
> >  25 KSP preconditioned resid norm 1.648500148983e-06 true resid norm 4.262645649931e-06 ||r(i)||/||b|| 2.657759561118e-13
> >  26 KSP preconditioned resid norm 7.373967102970e-07 true resid norm 1.906404712490e-06 ||r(i)||/||b|| 1.188643337515e-13
> >  27 KSP preconditioned resid norm 3.298179068243e-07 true resid norm 8.525291822193e-07 ||r(i)||/||b|| 5.315519447909e-14
> >  28 KSP preconditioned resid norm 1.475043181061e-07 true resid norm 3.812420047527e-07 ||r(i)||/||b|| 2.377043898189e-14
> >  29 KSP preconditioned resid norm 6.596561561066e-08 true resid norm 1.704561511399e-07 ||r(i)||/||b|| 1.062794101712e-14
> >  30 KSP preconditioned resid norm 2.949954993990e-08 true resid norm 7.626396764524e-08 ||r(i)||/||b|| 4.755058379793e-15
> >  31 KSP preconditioned resid norm 1.319299835423e-08 true resid norm 3.408580527641e-08 ||r(i)||/||b|| 2.125249957693e-15
> >  32 KSP preconditioned resid norm 5.894082812579e-09 true resid norm 1.548343581158e-08 ||r(i)||/||b|| 9.653922222659e-16
> >  33 KSP preconditioned resid norm 2.636703982134e-09 true resid norm 7.135606738493e-09 ||r(i)||/||b|| 4.449050798748e-16
> >  34 KSP preconditioned resid norm 1.180985878209e-09 true resid norm 3.381752613021e-09 ||r(i)||/||b|| 2.108522752938e-16
> >  35 KSP preconditioned resid norm 5.286215416700e-10 true resid norm 2.401714396480e-09 ||r(i)||/||b|| 1.497468925296e-16
> >  36 KSP preconditioned resid norm 2.343627265669e-10 true resid norm 2.406695615135e-09 ||r(i)||/||b|| 1.500574715125e-16
> >  37 KSP preconditioned resid norm 1.063481191780e-10 true resid norm 1.939409664821e-09 ||r(i)||/||b|| 1.209221925282e-16
> >  38 KSP preconditioned resid norm 4.641441861184e-11 true resid norm 2.137293190758e-09 ||r(i)||/||b|| 1.332602303628e-16
> >  39 KSP preconditioned resid norm 2.197549316276e-11 true resid norm 2.134629170008e-09 ||r(i)||/||b|| 1.330941286692e-16
> >  40 KSP preconditioned resid norm 1.014465992249e-11 true resid norm 2.134073377634e-09 ||r(i)||/||b|| 1.330594750147e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 40
> > 
> > 3) And lastly with chebyshev/jacobi:
> > 
> > -ksp_type chebyshev
> > -ksp_rtol 1e-15
> > -ksp_monitor_true_residual
> > -pc_type jacobi
> > 
> >   0 KSP preconditioned resid norm 1.124259077563e+04 true resid norm 2.745522604971e+06 ||r(i)||/||b|| 1.711833343159e-01
> >   1 KSP preconditioned resid norm 7.344319020428e+03 true resid norm 7.783641348970e+06 ||r(i)||/||b|| 4.853100378135e-01
> >   2 KSP preconditioned resid norm 1.071669918360e+04 true resid norm 2.799860726937e+06 ||r(i)||/||b|| 1.745713162185e-01
> >   3 KSP preconditioned resid norm 3.419051822673e+03 true resid norm 1.775069259453e+06 ||r(i)||/||b|| 1.106755682597e-01
> >   4 KSP preconditioned resid norm 4.986468711193e+03 true resid norm 1.206925036347e+06 ||r(i)||/||b|| 7.525177597052e-02
> >   5 KSP preconditioned resid norm 1.700832321100e+03 true resid norm 2.637405831602e+05 ||r(i)||/||b|| 1.644422535005e-02
> >   6 KSP preconditioned resid norm 1.643529813686e+03 true resid norm 3.974328566033e+05 ||r(i)||/||b|| 2.477993859417e-02
> >   7 KSP preconditioned resid norm 7.473371550560e+02 true resid norm 5.323098795195e+04 ||r(i)||/||b|| 3.318952096788e-03
> >   8 KSP preconditioned resid norm 4.683110030109e+02 true resid norm 1.087414808661e+05 ||r(i)||/||b|| 6.780031327884e-03
> >   9 KSP preconditioned resid norm 2.873339948815e+02 true resid norm 3.568238211189e+04 ||r(i)||/||b|| 2.224796523325e-03
> >  10 KSP preconditioned resid norm 1.262071076718e+02 true resid norm 2.462817350416e+04 ||r(i)||/||b|| 1.535566617052e-03
> >  11 KSP preconditioned resid norm 1.002027390320e+02 true resid norm 1.546159763209e+04 ||r(i)||/||b|| 9.640306117750e-04
> >  12 KSP preconditioned resid norm 3.354594608285e+01 true resid norm 4.591691194787e+03 ||r(i)||/||b|| 2.862919458211e-04
> >  13 KSP preconditioned resid norm 3.131257260705e+01 true resid norm 5.245042976548e+03 ||r(i)||/||b|| 3.270284293891e-04
> >  14 KSP preconditioned resid norm 9.505226922340e+00 true resid norm 1.166368684616e+03 ||r(i)||/||b|| 7.272308744166e-05
> >  15 KSP preconditioned resid norm 8.743145384463e+00 true resid norm 1.504382325425e+03 ||r(i)||/||b|| 9.379823793327e-05
> >  16 KSP preconditioned resid norm 3.088859985366e+00 true resid norm 5.407763762347e+02 ||r(i)||/||b|| 3.371740703775e-05
> >  17 KSP preconditioned resid norm 2.376841598522e+00 true resid norm 3.716033743805e+02 ||r(i)||/||b|| 2.316947037855e-05
> >  18 KSP preconditioned resid norm 1.081970394434e+00 true resid norm 2.205852224893e+02 ||r(i)||/||b|| 1.375348861385e-05
> >  19 KSP preconditioned resid norm 6.554707900903e-01 true resid norm 8.301134625673e+01 ||r(i)||/||b|| 5.175757435963e-06
> >  20 KSP preconditioned resid norm 3.637023760759e-01 true resid norm 7.510650297359e+01 ||r(i)||/||b|| 4.682890457559e-06
> >  21 KSP preconditioned resid norm 1.871804319271e-01 true resid norm 2.219347914083e+01 ||r(i)||/||b|| 1.383763423590e-06
> >  22 KSP preconditioned resid norm 1.100146172732e-01 true resid norm 2.219515834389e+01 ||r(i)||/||b|| 1.383868121901e-06
> >  23 KSP preconditioned resid norm 5.760669698705e-02 true resid norm 8.337640509358e+00 ||r(i)||/||b|| 5.198518854427e-07
> >  24 KSP preconditioned resid norm 2.957448587725e-02 true resid norm 5.874493674310e+00 ||r(i)||/||b|| 3.662746803708e-07
> >  25 KSP preconditioned resid norm 1.998893198438e-02 true resid norm 3.167578238976e+00 ||r(i)||/||b|| 1.974985030802e-07
> >  26 KSP preconditioned resid norm 8.664848450375e-03 true resid norm 1.486849213225e+00 ||r(i)||/||b|| 9.270504838826e-08
> >  27 KSP preconditioned resid norm 6.981883525312e-03 true resid norm 1.069480324023e+00 ||r(i)||/||b|| 6.668209816232e-08
> >  28 KSP preconditioned resid norm 2.719053601907e-03 true resid norm 4.199959208195e-01 ||r(i)||/||b|| 2.618674564719e-08
> >  29 KSP preconditioned resid norm 2.165577279425e-03 true resid norm 3.210144228918e-01 ||r(i)||/||b|| 2.001524925514e-08
> >  30 KSP preconditioned resid norm 7.988525722643e-04 true resid norm 1.420960576866e-01 ||r(i)||/||b|| 8.859689191379e-09
> >  31 KSP preconditioned resid norm 6.325404656692e-04 true resid norm 8.848430840431e-02 ||r(i)||/||b|| 5.516996625657e-09
> >  32 KSP preconditioned resid norm 2.774874251260e-04 true resid norm 4.978943834544e-02 ||r(i)||/||b|| 3.104371478953e-09
> >  33 KSP preconditioned resid norm 2.189482639986e-04 true resid norm 2.363450483074e-02 ||r(i)||/||b|| 1.473611375302e-09
> >  34 KSP preconditioned resid norm 1.083040043835e-04 true resid norm 1.640956513288e-02 ||r(i)||/||b|| 1.023136385414e-09
> >  35 KSP preconditioned resid norm 7.862356661381e-05 true resid norm 6.670818331105e-03 ||r(i)||/||b|| 4.159255226918e-10
> >  36 KSP preconditioned resid norm 3.874849522187e-05 true resid norm 5.033219998314e-03 ||r(i)||/||b|| 3.138212667043e-10
> >  37 KSP preconditioned resid norm 2.528412836894e-05 true resid norm 2.108237735582e-03 ||r(i)||/||b|| 1.314486227337e-10
> >  38 KSP preconditioned resid norm 1.267202237267e-05 true resid norm 1.451831066002e-03 ||r(i)||/||b|| 9.052166691026e-11
> >  39 KSP preconditioned resid norm 8.210280946453e-06 true resid norm 7.040263504011e-04 ||r(i)||/||b|| 4.389604292084e-11
> >  40 KSP preconditioned resid norm 4.663490696194e-06 true resid norm 4.017228546553e-04 ||r(i)||/||b|| 2.504741997254e-11
> >  41 KSP preconditioned resid norm 3.031143852348e-06 true resid norm 2.322717593194e-04 ||r(i)||/||b|| 1.448214418476e-11
> >  42 KSP preconditioned resid norm 1.849864051869e-06 true resid norm 1.124281997627e-04 ||r(i)||/||b|| 7.009898250945e-12
> >  43 KSP preconditioned resid norm 1.124434187023e-06 true resid norm 7.306110293564e-05 ||r(i)||/||b|| 4.555359765270e-12
> >  44 KSP preconditioned resid norm 6.544412722416e-07 true resid norm 3.401751796431e-05 ||r(i)||/||b|| 2.120992243786e-12
> >  45 KSP preconditioned resid norm 3.793047150173e-07 true resid norm 2.148969752882e-05 ||r(i)||/||b|| 1.339882640108e-12
> >  46 KSP preconditioned resid norm 2.171588698514e-07 true resid norm 1.106350941000e-05 ||r(i)||/||b|| 6.898098112943e-13
> >  47 KSP preconditioned resid norm 1.296462934907e-07 true resid norm 6.099370095521e-06 ||r(i)||/||b|| 3.802957252247e-13
> >  48 KSP preconditioned resid norm 8.025171649527e-08 true resid norm 3.573262429072e-06 ||r(i)||/||b|| 2.227929123173e-13
> >  49 KSP preconditioned resid norm 4.871794377896e-08 true resid norm 1.812080394338e-06 ||r(i)||/||b|| 1.129832125183e-13
> >  50 KSP preconditioned resid norm 3.113615807350e-08 true resid norm 1.077814343384e-06 ||r(i)||/||b|| 6.720172426917e-14
> >  51 KSP preconditioned resid norm 1.796713291999e-08 true resid norm 5.820635497903e-07 ||r(i)||/||b|| 3.629166230738e-14
> >  52 KSP preconditioned resid norm 1.101194810402e-08 true resid norm 3.130702427551e-07 ||r(i)||/||b|| 1.951992962392e-14
> >  53 KSP preconditioned resid norm 6.328675584656e-09 true resid norm 1.886022841038e-07 ||r(i)||/||b|| 1.175935240673e-14
> >  54 KSP preconditioned resid norm 3.753197413786e-09 true resid norm 9.249216284944e-08 ||r(i)||/||b|| 5.766886350159e-15
> >  55 KSP preconditioned resid norm 2.289205545523e-09 true resid norm 5.644941358874e-08 ||r(i)||/||b|| 3.519620935120e-15
> >  56 KSP preconditioned resid norm 1.398041051045e-09 true resid norm 2.841582922959e-08 ||r(i)||/||b|| 1.771726951389e-15
> >  57 KSP preconditioned resid norm 8.587888866230e-10 true resid norm 1.829052851330e-08 ||r(i)||/||b|| 1.140414452112e-15
> >  58 KSP preconditioned resid norm 5.353939794444e-10 true resid norm 1.100004853784e-08 ||r(i)||/||b|| 6.858530259177e-16
> >  59 KSP preconditioned resid norm 3.152419669065e-10 true resid norm 6.833334353224e-09 ||r(i)||/||b|| 4.260583966647e-16
> >  60 KSP preconditioned resid norm 1.930837697706e-10 true resid norm 3.532215487724e-09 ||r(i)||/||b|| 2.202336355258e-16
> >  61 KSP preconditioned resid norm 1.138921366053e-10 true resid norm 2.879312701518e-09 ||r(i)||/||b|| 1.795251468306e-16
> >  62 KSP preconditioned resid norm 6.820698934300e-11 true resid norm 2.528012115752e-09 ||r(i)||/||b|| 1.576215553214e-16
> >  63 KSP preconditioned resid norm 4.141390392052e-11 true resid norm 3.265136945688e-09 ||r(i)||/||b|| 2.035812884400e-16
> >  64 KSP preconditioned resid norm 2.447449492240e-11 true resid norm 3.548082053472e-09 ||r(i)||/||b|| 2.212229158996e-16
> >  65 KSP preconditioned resid norm 1.530621705437e-11 true resid norm 2.329307411648e-09 ||r(i)||/||b|| 1.452323170280e-16
> >  66 KSP preconditioned resid norm 1.110145418759e-11 true resid norm 2.794373041066e-09 ||r(i)||/||b|| 1.742291590046e-16
> > Linear solve converged due to CONVERGED_RTOL iterations 67
> > 
> > Looks like neither of these ksp/pc combos are good? I also tried -pc_gamg_agg_nsmooths 0 but it didn't improve the solver at all. Here's the GAMG info I was able to grep from the out-of-box params:
> > 
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.587728e+00 min=2.394056e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=5, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 5.21674 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 537 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.939736e+00 min=6.783380e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 2) N=537, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 9.85289 nnz ave. (N=537)
> > [0] PCGAMGProlongator_AGG(): New grid 100 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.521731e+00 min=5.974776e-02 PC=jacobi
> > [0] PCSetUp_GAMG(): 3) N=100, n data cols=1, nnz/row (ave)=12, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 12.4 nnz ave. (N=100)
> > [0] PCGAMGProlongator_AGG(): New grid 17 nodes
> > [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.560264e+00 min=4.842076e-01 PC=jacobi
> > [0] PCSetUp_GAMG(): 4) N=17, n data cols=1, nnz/row (ave)=9, 1 active pes
> > [0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.31821
> > 
> > And the one with pc_gamg_agg_nsmooths 0
> > 
> > [0] PCSetUp_GAMG(): level 0) N=8541, n data rows=1, n data cols=1, nnz/row (ave)=5, np=1
> > [0] PCGAMGFilterGraph():       99.9114% nnz after filtering, with threshold 0., 5.42079 nnz ave. (N=8541)
> > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> > [0] PCGAMGProlongator_AGG(): New grid 1541 nodes
> > [0] PCSetUp_GAMG(): 1) N=1541, n data cols=1, nnz/row (ave)=3, 1 active pes
> > [0] PCGAMGFilterGraph():       99.7467% nnz after filtering, with threshold 0., 3.07398 nnz ave. (N=1541)
> > [0] PCGAMGProlongator_AGG(): New grid 814 nodes
> > [0] PCSetUp_GAMG(): 2) N=814, n data cols=1, nnz/row (ave)=3, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.02211 nnz ave. (N=814)
> > [0] PCGAMGProlongator_AGG(): New grid 461 nodes
> > [0] PCSetUp_GAMG(): 3) N=461, n data cols=1, nnz/row (ave)=3, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3.00434 nnz ave. (N=461)
> > [0] PCGAMGProlongator_AGG(): New grid 290 nodes
> > [0] PCSetUp_GAMG(): 4) N=290, n data cols=1, nnz/row (ave)=3, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=290)
> > [0] PCGAMGProlongator_AGG(): New grid 197 nodes
> > [0] PCSetUp_GAMG(): 5) N=197, n data cols=1, nnz/row (ave)=3, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 3. nnz ave. (N=197)
> > [0] PCGAMGProlongator_AGG(): New grid 127 nodes
> > [0] PCSetUp_GAMG(): 6) N=127, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.98425 nnz ave. (N=127)
> > [0] PCGAMGProlongator_AGG(): New grid 82 nodes
> > [0] PCSetUp_GAMG(): 7) N=82, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.97561 nnz ave. (N=82)
> > [0] PCGAMGProlongator_AGG(): New grid 66 nodes
> > [0] PCSetUp_GAMG(): 8) N=66, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCGAMGFilterGraph():       100.% nnz after filtering, with threshold 0., 2.9697 nnz ave. (N=66)
> > [0] PCGAMGProlongator_AGG(): New grid 36 nodes
> > [0] PCSetUp_GAMG(): 9) N=36, n data cols=1, nnz/row (ave)=2, 1 active pes
> > [0] PCSetUp_GAMG(): 10 levels, grid complexity = 1.23689
> > 
> > 
> > Thanks,
> > Justin
> > 
> > On Fri, Feb 1, 2019 at 7:13 AM Mark Adams <mfadams at lbl.gov> wrote:
> > 
> > 
> > Both GAMG and ILU are nice and dandy for this,
> > 
> > I would test Richardson/SOR and Chebyshev/Jacobi on the tiny system and converge it way down, say rtol = 1.e-12. See which one is better in the early iteration and pick it. It would be nice to check that it solves the problem ...
> > 
> > The residual drops about 5 orders in the first iteration and then flatlines. That is very bad. Check that the smoother can actually solve the problem.  
> >  
> > but as soon as I look at a bigger system, like a network with 8500 buses, the out-of-box gamg craps out. I am not sure where to start when it comes to tuning GAMG.
> > 
> > First try -pc_gamg_nsmooths 0
> > 
> > You can run with -info and grep on GAMG to see some diagnostic output. Eigen estimates are fragile in practice and with this parameter and SOR there are no eigen estimates needed. The max eigen values for all levels should be between say 2 (or less) and say 3-4. Much higher is a sign of a problem.
> >  
> > Attached is the ksp monitor/view output for gamg on the unsuccessful solve
> > 
> > I'm also attaching a zip file which contains the simple PETSc script that loads the binary matrix/vector as well as two test cases, if you guys are interested in trying it out. It only works if you have PETSc configured with complex numbers.
> > 
> > Thanks
> > 
> > Justin
> > 
> > PS - A couple years ago I had asked if there was a paper/tutorial on using/tuning GAMG. Does such a thing exist today?
> > 
> > There is a write up in the manual that is tutorial like.
> >  
> > 
> > On Thu, Jan 31, 2019 at 5:00 PM Matthew Knepley <knepley at gmail.com> wrote:
> > On Thu, Jan 31, 2019 at 6:22 PM Justin Chang <jychang48 at gmail.com> wrote:
> > Here's IMHO the simplest explanation of the equations I'm trying to solve:
> > 
> > http://home.eng.iastate.edu/~jdm/ee458_2011/PowerFlowEquations.pdf
> > 
> > Right now we're just trying to solve eq(5) (in section 1), inverting the linear Y-bus matrix. Eventually we have to be able to solve equations like those in the next section.
> > 
> > Maybe I am reading this wrong, but the Y-bus matrix looks like an M-matrix to me (if all the y's are positive). This means
> > that it should be really easy to solve, and I think GAMG should do it. You can start out just doing relaxation, like SOR, on
> > small examples.
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > On Thu, Jan 31, 2019 at 1:47 PM Matthew Knepley <knepley at gmail.com> wrote:
> > On Thu, Jan 31, 2019 at 3:20 PM Justin Chang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > Hi all,
> > 
> > I'm working with some folks to extract a linear system of equations from an external software package that solves power flow equations in complex form. Since that external package uses serial direct solvers like KLU from suitesparse, I want a proof-of-concept where the same matrix can be solved in PETSc using its parallel solvers. 
> > 
> > I got mumps to achieve a very minor speedup across two MPI processes on a single node (went from solving a 300k dog system in 1.8 seconds to 1.5 seconds). However I want to use iterative solvers and preconditioners but I have never worked with complex numbers so I am not sure what the "best" options are given PETSc's capabilities.
> > 
> > So far I tried GMRES/BJACOBI and it craps out (unsurprisingly). I believe I also tried BICG with BJACOBI and while it did converge it converged slowly. Does anyone have recommendations on how one would go about preconditioning PETSc matrices with complex numbers? I was originally thinking about converting it to cartesian form: Declaring all voltages = sqrt(real^2+imaginary^2) and all angles to be something like a conditional arctan(imaginary/real) because all the papers I've seen in literature that claim to successfully precondition power flow equations operate in this form.
> > 
> > 1) We really need to see the (simplified) equations
> > 
> > 2) All complex equations can be converted to a system of real equations twice as large, but this is not necessarily the best way to go
> > 
> >  Thanks,
> > 
> >     Matt
> >  
> > Justin
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> 


From jed at jedbrown.org  Mon Feb 11 21:26:11 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 Feb 2019 20:26:11 -0700
Subject: [petsc-users] Preconditioning systems of equations with complex
 numbers
In-Reply-To: <CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>
References: <CAP2=TMhmQ0A_M=tmo98nYNzHeOyLCJ2Dcy_vBj5rzktfhYNGaw@mail.gmail.com>
	<CAMYG4G=eGbNy0wsnFh+B1Fi0Hxm88YPvcxtQcp9xocV=BHJWdA@mail.gmail.com>
	<CAP2=TMibFdRJpemkCRH2vPdS13nMTBZXGCPs50c8=SMnQfTgfA@mail.gmail.com>
	<CAMYG4G=_8rSHiDti4xnW9pf4zShy=XZc+pnBr_zP6fK9sbUBxQ@mail.gmail.com>
	<CAP2=TMiD6w7yV-kwKOxVvJ1-h-KFrp3tq6Ja=URb=XFcy8-TJA@mail.gmail.com>
	<CADOhEh5aQjiKTm2JNsm1OvGzAqnboF43FT0cethFcgZsptLycw@mail.gmail.com>
	<CAP2=TMi9WvrZ4295DHGjyDQ5SJPP1tsHFVdVSEmCCb_sJ=bgaA@mail.gmail.com>
	<CAMYG4G=vx4wTEw7y3tRGjwgFodFuwFYVuK-MVO4pL90iPexA=g@mail.gmail.com>
	<CAP2=TMgLeHjai5DjEPNmqtHu70aNM2_6HUmUYnMHAcrXRP=HPA@mail.gmail.com>
	<CAMYG4GnBLJOsCand-zYmWbPsZ4aQOsBp-XaOiJO8Xm7iHpR74g@mail.gmail.com>
	<CAP2=TMjRbCnCrW84uBGvReG4s1L0+HXDv4MDQqzXJraviL7qLQ@mail.gmail.com>
	<8E6D7031-2C03-4FA0-B82F-CE79BE699F7D@anl.gov>
	<CADOhEh5HJ54VS2S_nfzd4wJAKQBQaRTDvoF1R7o+wBn_+28waQ@mail.gmail.com>
	<CAP2=TMh5Wd41oycpPPU5LHS99aO6=7h0C7CuA8FAQk7z2u=vEg@mail.gmail.com>
	<DM6PR09MB32447943CC77CBCB7F36A58DE76F0@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMhrRYCG6P4v9sFFY-yqMt-LF9xa+0k05r98_tO+w1YUYA@mail.gmail.com>
	<DM6PR09MB324407530365188B97EA9E30E7680@DM6PR09MB3244.namprd09.prod.outlook.com>
	<CAP2=TMiwuO+ysCwqfOM=o_g4t_pzm1a-5rJXv04MkJAJhxgOCQ@mail.gmail.com>
Message-ID: <87y36ld6wc.fsf@jedbrown.org>

Justin Chang via petsc-users <petsc-users at mcs.anl.gov> writes:

> So I used -mat_view draw -draw_pause -1 on my medium sized matrix and got
> this output:
>
> [image: 1MPI.png]
>
> So it seems there are lots of off-diagonal terms, and that a decomposition
> of the problem via matload would give a terrible unbalanced problem.
>
> Given the initial A and b Mat/Vec, I experimented with MatPartioning and
> inserted the following lines into my code:
>
> Mat               Apart;
> Vec               bpart;
> MatPartitioning   part;
> IS                is,isrows;
> ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part);CHKERRQ(ierr);
> ierr = MatPartitioningSetAdjacency(part, A);CHKERRQ(ierr);
> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
> ierr = MatPartitioningApply(part, &is);CHKERRQ(ierr);
> ierr = ISBuildTwoSided(is,NULL,&isrows);CHKERRQ(ierr);
> ierr = MatCreateSubMatrix(A, isrows,isrows, MAT_INITIAL_MATRIX,
> &Apart);CHKERRQ(ierr);
> ierr = MatSetOptionsPrefix(Apart, "part_");CHKERRQ(ierr);
> ierr = MatSetFromOptions(Apart);CHKERRQ(ierr);
> ierr = VecGetSubVector(b,isrows,&bpart);CHKERRQ(ierr);
>
> /* Set Apart and bpart in the KSPSolve */
> ...
>
> And here are the mat_draw figures from 2 and 4 MPI processes respectively:
>
> [image: 2MPI.png][image: 4MPI.png]
>
> Is this "right"? It just feels like I'm just duplicating the nnz structure
> among all the MPI processes. And it didn't really improve the performance
> of ASM.

ASM might not be an appropriate preconditioner (or it might need a
special sort of overlap for stability of the local problems).  The edge
cuts look relatively small, so it doesn't look to me like power law or
social network problems that don't admit vertex partitions with low edge
cut.

We really have to understand the spectrum to comment further on fast
solvers.

From eda.oktay at metu.edu.tr  Tue Feb 12 01:50:05 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 12 Feb 2019 10:50:05 +0300
Subject: [petsc-users] Problem in loading Matrix Market format
Message-ID: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>

Hello,

I am trying to load matrix in Matrix Market format. I found an example on
mat file  (ex78) whih can be tested by using .dat file. Since .dat file and
.mtx file are similar  in structure (specially afiro_A.dat file is similar
to amesos2_test_mat0.mtx since they both have 3 columns and the columns
represent the same properties), I tried to run ex78 by using
amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error
"Badly formatted input file". Here is the full error message:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Badly formatted input file

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr
by edaoktay Tue Feb 12 10:47:58 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 73 in
/home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -Ain
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

I know there is also an example (ex72) for Matrix Market format but in
description, it is only proper for symmmetric and lower triangle, so I
decided to use ex78.

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/360f5d38/attachment.html>

From jroman at dsic.upv.es  Tue Feb 12 03:05:13 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 12 Feb 2019 10:05:13 +0100
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
Message-ID: <9335798C-7B6C-4B94-983C-D9F6EBF125D5@dsic.upv.es>

It is better to convert the matrices to PETSc binary format first. One easy way is to read them into Matlab with mmread.m and write with PETSc's PetscBinaryWrite.m. This can be done similarly in python.
Jose


> El 12 feb 2019, a las 8:50, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> 
> Hello,
> 
> I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Badly formatted input file
> 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> [0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
> :
> system msg for write_line failure : Bad file descriptor
> 
> I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.
> 
> Best regards,
> 
> Eda 


From jczhang at mcs.anl.gov  Tue Feb 12 09:26:53 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 12 Feb 2019 15:26:53 +0000
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
Message-ID: <CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>

Eda,
  I have a code that can read in Matrix Market and write out PETSc binary files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout <outfile>.  You can have a try.
--Junchao Zhang


On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello,

I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Badly formatted input file

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr<http://7330.wls.metu.edu.tr> by edaoktay Tue Feb 12 10:47:58 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/b1c8697a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: matrixmarket2petsc.tgz
Type: application/gzip
Size: 5873 bytes
Desc: matrixmarket2petsc.tgz
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/b1c8697a/attachment-0001.bin>

From knepley at gmail.com  Tue Feb 12 09:46:37 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 12 Feb 2019 10:46:37 -0500
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
Message-ID: <CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>

Hi Junchao,

Could you fix the MM example in PETSc to have this full support? That way
we will always have it.

 Thanks,

    Matt

On Tue, Feb 12, 2019 at 10:27 AM Zhang, Junchao via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Eda,
>   I have a code that can read in Matrix Market and write out PETSc binary
> files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout <outfile>.  You
> can have a try.
> --Junchao Zhang
>
>
> On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello,
>>
>> I am trying to load matrix in Matrix Market format. I found an example on
>> mat file  (ex78) whih can be tested by using .dat file. Since .dat file and
>> .mtx file are similar  in structure (specially afiro_A.dat file is similar
>> to amesos2_test_mat0.mtx since they both have 3 columns and the columns
>> represent the same properties), I tried to run ex78 by using
>> amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error
>> "Badly formatted input file". Here is the full error message:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Badly formatted input file
>>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named
>> 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> [0]PETSC ERROR: #1 main() line 73 in
>> /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -Ain
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
>> :
>> system msg for write_line failure : Bad file descriptor
>>
>> I know there is also an example (ex72) for Matrix Market format but in
>> description, it is only proper for symmmetric and lower triangle, so I
>> decided to use ex78.
>>
>> Best regards,
>>
>> Eda
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/14f4ad7d/attachment.html>

From jczhang at mcs.anl.gov  Tue Feb 12 09:51:52 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 12 Feb 2019 15:51:52 +0000
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
	<CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>
Message-ID: <CA+MQGp9qd_9rTPfTNz6WuqOXPf9PX6j-tk+5Bf=rHAK4GEaFaQ@mail.gmail.com>

Sure.
--Junchao Zhang


On Tue, Feb 12, 2019 at 9:47 AM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
Hi Junchao,

Could you fix the MM example in PETSc to have this full support? That way we will always have it.

 Thanks,

    Matt

On Tue, Feb 12, 2019 at 10:27 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Eda,
  I have a code that can read in Matrix Market and write out PETSc binary files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout <outfile>.  You can have a try.
--Junchao Zhang


On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello,

I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Badly formatted input file

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr<http://7330.wls.metu.edu.tr> by edaoktay Tue Feb 12 10:47:58 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.

Best regards,

Eda


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/ba5e0a17/attachment.html>

From hzhang at mcs.anl.gov  Tue Feb 12 10:09:45 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 12 Feb 2019 16:09:45 +0000
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+MQGp9qd_9rTPfTNz6WuqOXPf9PX6j-tk+5Bf=rHAK4GEaFaQ@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
	<CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>
	<CA+MQGp9qd_9rTPfTNz6WuqOXPf9PX6j-tk+5Bf=rHAK4GEaFaQ@mail.gmail.com>
Message-ID: <CAGCphBuRNHkiUmm9NkQ9O55esa-SRLVc+gfbs-Nrk7m+bn73aA@mail.gmail.com>

We have /home/petsc/datafiles/matrices/MtxMarket/mm2petsc.c
Hong

On Tue, Feb 12, 2019 at 9:52 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Sure.
--Junchao Zhang


On Tue, Feb 12, 2019 at 9:47 AM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
Hi Junchao,

Could you fix the MM example in PETSc to have this full support? That way we will always have it.

 Thanks,

    Matt

On Tue, Feb 12, 2019 at 10:27 AM Zhang, Junchao via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Eda,
  I have a code that can read in Matrix Market and write out PETSc binary files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout <outfile>.  You can have a try.
--Junchao Zhang


On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello,

I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Badly formatted input file

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr<http://7330.wls.metu.edu.tr> by edaoktay Tue Feb 12 10:47:58 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.

Best regards,

Eda


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/497c75b4/attachment-0001.html>

From knepley at gmail.com  Tue Feb 12 15:47:14 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 12 Feb 2019 16:47:14 -0500
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CAGCphBuRNHkiUmm9NkQ9O55esa-SRLVc+gfbs-Nrk7m+bn73aA@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
	<CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>
	<CA+MQGp9qd_9rTPfTNz6WuqOXPf9PX6j-tk+5Bf=rHAK4GEaFaQ@mail.gmail.com>
	<CAGCphBuRNHkiUmm9NkQ9O55esa-SRLVc+gfbs-Nrk7m+bn73aA@mail.gmail.com>
Message-ID: <CAMYG4Gm2341=uPBSxAn-SZ7yOPh6O2jmJ45Oy5+PkWQC-_9_ag@mail.gmail.com>

It definitely should not be there under 'datafiles'. We should put it in an
example, as Junchao graciously agreed to do.

  Thanks,

    Matt

On Tue, Feb 12, 2019 at 11:15 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> We have /home/petsc/datafiles/matrices/MtxMarket/mm2petsc.c
> Hong
>
> On Tue, Feb 12, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Sure.
>> --Junchao Zhang
>>
>>
>> On Tue, Feb 12, 2019 at 9:47 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> Hi Junchao,
>>>
>>> Could you fix the MM example in PETSc to have this full support? That
>>> way we will always have it.
>>>
>>>  Thanks,
>>>
>>>     Matt
>>>
>>> On Tue, Feb 12, 2019 at 10:27 AM Zhang, Junchao via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Eda,
>>>>   I have a code that can read in Matrix Market and write out PETSc
>>>> binary files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout
>>>> <outfile>.  You can have a try.
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am trying to load matrix in Matrix Market format. I found an example
>>>>> on mat file  (ex78) whih can be tested by using .dat file. Since .dat file
>>>>> and .mtx file are similar  in structure (specially afiro_A.dat file is
>>>>> similar to amesos2_test_mat0.mtx since they both have 3 columns and the
>>>>> columns represent the same properties), I tried to run ex78 by using
>>>>> amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error
>>>>> "Badly formatted input file". Here is the full error message:
>>>>>
>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Badly formatted input file
>>>>>
>>>>> [0]PETSC ERROR: See
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>> shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>> [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named
>>>>> 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
>>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>> --download-slepc --download-mpich
>>>>> [0]PETSC ERROR: #1 main() line 73 in
>>>>> /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
>>>>> [0]PETSC ERROR: PETSc Option Table entries:
>>>>> [0]PETSC ERROR: -Ain
>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
>>>>> :
>>>>> system msg for write_line failure : Bad file descriptor
>>>>>
>>>>> I know there is also an example (ex72) for Matrix Market format but in
>>>>> description, it is only proper for symmmetric and lower triangle, so I
>>>>> decided to use ex78.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Eda
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/f5a90a25/attachment.html>

From jed at jedbrown.org  Tue Feb 12 16:43:23 2019
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 12 Feb 2019 15:43:23 -0700
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CAMYG4Gm2341=uPBSxAn-SZ7yOPh6O2jmJ45Oy5+PkWQC-_9_ag@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<CA+MQGp9c8D-wT-7h4Sc7HNYJnYZ0j6kzN-UdHQQDiksvS7+Kdw@mail.gmail.com>
	<CAMYG4G=aEnmtXR2QtfXZZf+LJ8ymjGTxRgzKt73Pg2nqDv5JpA@mail.gmail.com>
	<CA+MQGp9qd_9rTPfTNz6WuqOXPf9PX6j-tk+5Bf=rHAK4GEaFaQ@mail.gmail.com>
	<CAGCphBuRNHkiUmm9NkQ9O55esa-SRLVc+gfbs-Nrk7m+bn73aA@mail.gmail.com>
	<CAMYG4Gm2341=uPBSxAn-SZ7yOPh6O2jmJ45Oy5+PkWQC-_9_ag@mail.gmail.com>
Message-ID: <87d0nwbpbo.fsf@jedbrown.org>

We should make the (two line) functionality a command-line feature of
PetscBinaryIO.py.  Then a user could do

  python -m PetscBinaryIO matrix.mm matrix.petsc

Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> writes:

> It definitely should not be there under 'datafiles'. We should put it in an
> example, as Junchao graciously agreed to do.
>
>   Thanks,
>
>     Matt
>
> On Tue, Feb 12, 2019 at 11:15 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
>> We have /home/petsc/datafiles/matrices/MtxMarket/mm2petsc.c
>> Hong
>>
>> On Tue, Feb 12, 2019 at 9:52 AM Zhang, Junchao via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Sure.
>>> --Junchao Zhang
>>>
>>>
>>> On Tue, Feb 12, 2019 at 9:47 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> Hi Junchao,
>>>>
>>>> Could you fix the MM example in PETSc to have this full support? That
>>>> way we will always have it.
>>>>
>>>>  Thanks,
>>>>
>>>>     Matt
>>>>
>>>> On Tue, Feb 12, 2019 at 10:27 AM Zhang, Junchao via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Eda,
>>>>>   I have a code that can read in Matrix Market and write out PETSc
>>>>> binary files.  Usage:  mpirun -n 1 ./mm2petsc -fin <infile> -fout
>>>>> <outfile>.  You can have a try.
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am trying to load matrix in Matrix Market format. I found an example
>>>>>> on mat file  (ex78) whih can be tested by using .dat file. Since .dat file
>>>>>> and .mtx file are similar  in structure (specially afiro_A.dat file is
>>>>>> similar to amesos2_test_mat0.mtx since they both have 3 columns and the
>>>>>> columns represent the same properties), I tried to run ex78 by using
>>>>>> amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error
>>>>>> "Badly formatted input file". Here is the full error message:
>>>>>>
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>> [0]PETSC ERROR: Badly formatted input file
>>>>>>
>>>>>> [0]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>>> [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named
>>>>>> 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
>>>>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>>>>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>>>>>> --download-metis --download-parmetis --download-superlu_dist
>>>>>> --download-slepc --download-mpich
>>>>>> [0]PETSC ERROR: #1 main() line 73 in
>>>>>> /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
>>>>>> [0]PETSC ERROR: PETSc Option Table entries:
>>>>>> [0]PETSC ERROR: -Ain
>>>>>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send
>>>>>> entire error message to petsc-maint at mcs.anl.gov----------
>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
>>>>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
>>>>>> :
>>>>>> system msg for write_line failure : Bad file descriptor
>>>>>>
>>>>>> I know there is also an example (ex72) for Matrix Market format but in
>>>>>> description, it is only proper for symmmetric and lower triangle, so I
>>>>>> decided to use ex78.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Eda
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

From maahi.buet at gmail.com  Tue Feb 12 17:42:22 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Tue, 12 Feb 2019 18:42:22 -0500
Subject: [petsc-users] Parallelization of the code
Message-ID: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>

Dear All,


I am tying to solve a linear system using ksp solvers. I have managed to
solve the system with a sequential code. The part of my sequential code
that deals with creating Matrix and setting values is as the following -

call MatCreate(PETSC_COMM_WORLD,Mp,ierr)
call MatSetSizes(Mp,PETSC_DECIDE,PETSC_DECIDE,u*v,u*v,ierr)
call MatSetFromOptions(Mp,ierr)
call MatSetUp(Mp,ierr)

Do p = 1,29008
Do r = 1,29008
if(Q(p,r)/=0.0) then
val(1) = Q(p,r)
col(1) = r-1
call MatSetValues(Mp,ione,p-1,ione,col,val,INSERT_VALUES,ierr)
endif
end Do
end Do

call MatAssemblyBegin(Mp,MAT_FINAL_ASSEMBLY,ierr)
call MatAssemblyEnd(Mp,MAT_FINAL_ASSEMBLY,ierr)

And the part of my sequential code that creates the vector is -

call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,u*v,Bx,ierr)
call VecSetFromOptions(Bx,ierr)
call VecDuplicate(Bx,Xp,ierr)
call VecSet(Bx,zero,ierr)

Do p = 1,29008
if(Fx(p,1)/=0.0) then
val(1) = Fx(p,1)
call VecSetValues(Bx,ione,p-1,val,INSERT_VALUES,ierr)
endif
end Do

call VecAssemblyBegin(Bx,ierr)
call VecAssemblyEnd(Bx,ierr)

So when I run the code on single processor, it runs fine. But when I tried
to run it on more than one processor, it failed.  Now from what I
understood from going through the manual is that if I use MatCreate to
create my Matrix, then depending on the no of processor  that I put in
after 'mpiexec -n ...' , it either acts either as a sequential code or a
parallel code. And I don't need to anything extra to make it work in
parallel, as PETSc does that internally.

So would you please let me know  what modifications I need to do to my
existing sequential code to make it work in parallel like using
MatGetOwnershipRange ?

Regards,
Maahi Talukder
MSc Student
Clarkson University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/708ad79d/attachment.html>

From knepley at gmail.com  Tue Feb 12 18:49:19 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 12 Feb 2019 19:49:19 -0500
Subject: [petsc-users] Parallelization of the code
In-Reply-To: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
References: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
Message-ID: <CAMYG4G=-YfqSiQnpXVV4N-o21rNG=AFGdBGon2moxTg0LxY+cQ@mail.gmail.com>

On Tue, Feb 12, 2019 at 6:43 PM Maahi Talukder via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
> Dear All,
>
>
> I am tying to solve a linear system using ksp solvers. I have managed to
> solve the system with a sequential code. The part of my sequential code
> that deals with creating Matrix and setting values is as the following -
>
> call MatCreate(PETSC_COMM_WORLD,Mp,ierr)
> call MatSetSizes(Mp,PETSC_DECIDE,PETSC_DECIDE,u*v,u*v,ierr)
> call MatSetFromOptions(Mp,ierr)
> call MatSetUp(Mp,ierr)
>
> Do p = 1,29008
> Do r = 1,29008
> if(Q(p,r)/=0.0) then
> val(1) = Q(p,r)
> col(1) = r-1
> call MatSetValues(Mp,ione,p-1,ione,col,val,INSERT_VALUES,ierr)
> endif
> end Do
> end Do
>
> call MatAssemblyBegin(Mp,MAT_FINAL_ASSEMBLY,ierr)
> call MatAssemblyEnd(Mp,MAT_FINAL_ASSEMBLY,ierr)
>
> And the part of my sequential code that creates the vector is -
>
> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,u*v,Bx,ierr)
> call VecSetFromOptions(Bx,ierr)
> call VecDuplicate(Bx,Xp,ierr)
> call VecSet(Bx,zero,ierr)
>
> Do p = 1,29008
> if(Fx(p,1)/=0.0) then
> val(1) = Fx(p,1)
> call VecSetValues(Bx,ione,p-1,val,INSERT_VALUES,ierr)
> endif
> end Do
>
> call VecAssemblyBegin(Bx,ierr)
> call VecAssemblyEnd(Bx,ierr)
>
> So when I run the code on single processor, it runs fine. But when I tried
> to run it on more than one processor, it failed.
>

How? We cannot guess how it fails. You have to send the ENTIRE error
message.

Looking about it should not fail, but it will be really inefficient because
every process sets every value in the matrix and vector.

   Matt


> Now from what I understood from going through the manual is that if I use
> MatCreate to create my Matrix, then depending on the no of processor  that
> I put in after 'mpiexec -n ...' , it either acts either as a sequential
> code or a parallel code. And I don't need to anything extra to make it work
> in parallel, as PETSc does that internally.
>
> So would you please let me know  what modifications I need to do to my
> existing sequential code to make it work in parallel like using
> MatGetOwnershipRange ?
>
> Regards,
> Maahi Talukder
> MSc Student
> Clarkson University
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190212/a81df7d5/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Feb 12 19:25:36 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 13 Feb 2019 01:25:36 +0000
Subject: [petsc-users] Parallelization of the code
In-Reply-To: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
References: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
Message-ID: <B0804670-3F12-456C-9814-9B5774C297DF@anl.gov>


  With MPI parallelism you need to do a decomposition of your data across the processes. Thus, for example, each process will generate a subset of the matrix entries. In addition, for large problems (which is what parallelism is for) you cannot use "dense" data structures, like your Q(), to store sparse matrices. For large problems with true sparse matrices the Q() will not fit on your machine and consists almost completely of zeros which it makes no sense to save in memory.

   How are your Q() entries being generated? This will determine the decomposition of the data you need to make. For example, with the finite element method each process would be assigned a subset of the elements (generally for a subregion of the entire domain).

   Barry



> On Feb 12, 2019, at 5:42 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> 
> Dear All,
> 
> 
> I am tying to solve a linear system using ksp solvers. I have managed to solve the system with a sequential code. The part of my sequential code that deals with creating Matrix and setting values is as the following - 
> 
> call MatCreate(PETSC_COMM_WORLD,Mp,ierr)
> call MatSetSizes(Mp,PETSC_DECIDE,PETSC_DECIDE,u*v,u*v,ierr)
> call MatSetFromOptions(Mp,ierr)
> call MatSetUp(Mp,ierr)
> 
> Do p = 1,29008
> Do r = 1,29008
> if(Q(p,r)/=0.0) then
> val(1) = Q(p,r)
> col(1) = r-1
> call MatSetValues(Mp,ione,p-1,ione,col,val,INSERT_VALUES,ierr)
> endif
> end Do
> end Do
> 
> call MatAssemblyBegin(Mp,MAT_FINAL_ASSEMBLY,ierr)
> call MatAssemblyEnd(Mp,MAT_FINAL_ASSEMBLY,ierr)
> 
> And the part of my sequential code that creates the vector is -
> 
> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,u*v,Bx,ierr)
> call VecSetFromOptions(Bx,ierr)
> call VecDuplicate(Bx,Xp,ierr)
> call VecSet(Bx,zero,ierr) 
> 
> Do p = 1,29008
> if(Fx(p,1)/=0.0) then
> val(1) = Fx(p,1)
> call VecSetValues(Bx,ione,p-1,val,INSERT_VALUES,ierr)
> endif
> end Do
> 
> call VecAssemblyBegin(Bx,ierr)
> call VecAssemblyEnd(Bx,ierr)
> 
> So when I run the code on single processor, it runs fine. But when I tried to run it on more than one processor, it failed.  Now from what I understood from going through the manual is that if I use MatCreate to create my Matrix, then depending on the no of processor  that I put in after 'mpiexec -n ...' , it either acts either as a sequential code or a parallel code. And I don't need to anything extra to make it work in parallel, as PETSc does that internally. 
> 
> So would you please let me know  what modifications I need to do to my existing sequential code to make it work in parallel like using MatGetOwnershipRange ? 
> 
> Regards,
> Maahi Talukder
> MSc Student
> Clarkson University
> 
> 
> 


From bsmith at mcs.anl.gov  Tue Feb 12 20:03:05 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 13 Feb 2019 02:03:05 +0000
Subject: [petsc-users] Parallelization of the code
In-Reply-To: <CAJOaupGgRriZMO5w6jRk8UEA=PPTq=nkSFgKJnMKAiqxNt2VXw@mail.gmail.com>
References: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
	<B0804670-3F12-456C-9814-9B5774C297DF@anl.gov>
	<CAJOaupGgRriZMO5w6jRk8UEA=PPTq=nkSFgKJnMKAiqxNt2VXw@mail.gmail.com>
Message-ID: <615AB6E4-4C56-43CB-80AF-F81071070C49@mcs.anl.gov>


  Sounds like you have a single structured grid so you should use https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDACreate2d.html and start with something like src/ksp/ksp/examples/tutorials/ex29.c and ex46.c 

  The DMDA manages dividing up the grid among processes and simplifies generating the matrix in parallel.

  Good luck,

   Barry


> On Feb 12, 2019, at 7:43 PM, Maahi Talukder <maahi.buet at gmail.com> wrote:
> 
> I am using Finite Difference to discretize my system and I am working with nine point stencil. So I am populating in Q() all the values. The code to do that is the following - 
> 
> Do i = 2,ymax-1
> 
> Do j = 2,xmax-1
> 
> g11(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i,j+1)-x(i,j-1)) +0.25*(y(i,j+1)-y(i,j-1))*(y(i,j+1)-y(i,j-1))
> g12(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i+1,j)-x(i-1,j))+ 0.25*(y(i,j+1)-y(i,j-1))*(y(i+1,j)-y(i-1,j))
> g22(i-1,j-1) = 0.25*(x(i+1,j)-x(i-1,j))*(x(i+1,j)-x(i-1,j)) + 0.25*(y(i+1,j)-y(i-1,j))*(y(i+1,j)-y(i-1,j))
> 
> M(j-1,j-1+xmax*(i-2))= (-1)*(g12(i-1,j-1)/2)
> M(j-1,j+xmax*(i-2))= g11(i-1,j-1)
> M(j-1,j+1+xmax*(i-2)) = g12(i-1,j-1)/2
> M(j-1,j-1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> M(j-1,j+xmax*(i-2)+xmax)= (-2)*(g22(i-1,j-1)+g11(i-1,j-1))
> M(j-1,j+1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> M(j-1,j-1+xmax*(i-2)+2*xmax) = g12(i-1,j-1)/2
> M(j-1,j+xmax*(i-2)+2*xmax) = g11(i-1,j-1)
> M(j-1,j+1+xmax*(i-2)+2*xmax) = (-1)*(g12(i-1,j-1)/2)       
> 
> end Do
> 
> N(1+(xmax-2)*(i-2):(xmax-2)*(i-1),1:xmax*ymax)= M(1:xmax-2,1:xmax*ymax)
> M = 0
> 
> end Do
> 
> E(1:(xmax-2)*(ymax-2),1:xmax*ymax-2*xmax) = N(1:(xmax-2)*(ymax-2),xmax+1:xmax*ymax-xmax)
> 
> Do i = 1,ymax-2
>     
> Q(1:(xmax-2)*(ymax-2),1+(xmax-2)*(i-1):(xmax-2)*i)= E(1:(xmax-2)*(ymax-2),2+xmax*(i-1):(xmax-1)+xmax*(i-1))
>     
> end Do
> 
> So how do I go about decomposing this Q() matrix across processes ? What PETSc function do I use for that?
> 
> 
> Regards,
> Maahi
> 
> On Tue, Feb 12, 2019 at 8:25 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   With MPI parallelism you need to do a decomposition of your data across the processes. Thus, for example, each process will generate a subset of the matrix entries. In addition, for large problems (which is what parallelism is for) you cannot use "dense" data structures, like your Q(), to store sparse matrices. For large problems with true sparse matrices the Q() will not fit on your machine and consists almost completely of zeros which it makes no sense to save in memory.
> 
>    How are your Q() entries being generated? This will determine the decomposition of the data you need to make. For example, with the finite element method each process would be assigned a subset of the elements (generally for a subregion of the entire domain).
> 
>    Barry
> 
> 
> 
> > On Feb 12, 2019, at 5:42 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > 
> > Dear All,
> > 
> > 
> > I am tying to solve a linear system using ksp solvers. I have managed to solve the system with a sequential code. The part of my sequential code that deals with creating Matrix and setting values is as the following - 
> > 
> > call MatCreate(PETSC_COMM_WORLD,Mp,ierr)
> > call MatSetSizes(Mp,PETSC_DECIDE,PETSC_DECIDE,u*v,u*v,ierr)
> > call MatSetFromOptions(Mp,ierr)
> > call MatSetUp(Mp,ierr)
> > 
> > Do p = 1,29008
> > Do r = 1,29008
> > if(Q(p,r)/=0.0) then
> > val(1) = Q(p,r)
> > col(1) = r-1
> > call MatSetValues(Mp,ione,p-1,ione,col,val,INSERT_VALUES,ierr)
> > endif
> > end Do
> > end Do
> > 
> > call MatAssemblyBegin(Mp,MAT_FINAL_ASSEMBLY,ierr)
> > call MatAssemblyEnd(Mp,MAT_FINAL_ASSEMBLY,ierr)
> > 
> > And the part of my sequential code that creates the vector is -
> > 
> > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,u*v,Bx,ierr)
> > call VecSetFromOptions(Bx,ierr)
> > call VecDuplicate(Bx,Xp,ierr)
> > call VecSet(Bx,zero,ierr) 
> > 
> > Do p = 1,29008
> > if(Fx(p,1)/=0.0) then
> > val(1) = Fx(p,1)
> > call VecSetValues(Bx,ione,p-1,val,INSERT_VALUES,ierr)
> > endif
> > end Do
> > 
> > call VecAssemblyBegin(Bx,ierr)
> > call VecAssemblyEnd(Bx,ierr)
> > 
> > So when I run the code on single processor, it runs fine. But when I tried to run it on more than one processor, it failed.  Now from what I understood from going through the manual is that if I use MatCreate to create my Matrix, then depending on the no of processor  that I put in after 'mpiexec -n ...' , it either acts either as a sequential code or a parallel code. And I don't need to anything extra to make it work in parallel, as PETSc does that internally. 
> > 
> > So would you please let me know  what modifications I need to do to my existing sequential code to make it work in parallel like using MatGetOwnershipRange ? 
> > 
> > Regards,
> > Maahi Talukder
> > MSc Student
> > Clarkson University
> > 
> > 
> > 
> 


From bsmith at mcs.anl.gov  Tue Feb 12 20:51:30 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 13 Feb 2019 02:51:30 +0000
Subject: [petsc-users] Parallelization of the code
In-Reply-To: <CAJOaupFv21OL=bqt8w7yMrP_EFRPx7Akvq5+eEDk9vDzhN3L5A@mail.gmail.com>
References: <CAJOaupHwEmrDaDi80CzvF5JS9wJHhPJJ-aY3mGc0+ZGcJ0-aqA@mail.gmail.com>
	<B0804670-3F12-456C-9814-9B5774C297DF@anl.gov>
	<CAJOaupGgRriZMO5w6jRk8UEA=PPTq=nkSFgKJnMKAiqxNt2VXw@mail.gmail.com>
	<615AB6E4-4C56-43CB-80AF-F81071070C49@mcs.anl.gov>
	<CAJOaupFv21OL=bqt8w7yMrP_EFRPx7Akvq5+eEDk9vDzhN3L5A@mail.gmail.com>
Message-ID: <AC4E0921-52FE-4CBC-8FEF-79990FCE7A12@mcs.anl.gov>



> On Feb 12, 2019, at 8:29 PM, Maahi Talukder <maahi.buet at gmail.com> wrote:
> 
> Thank you for your suggestions. I will go through them.
> 
> But can't I do it any other way? Like using MatMPIAIJPreallocation? Cause I have already calculated the elements of the matrix, and all I have to do is to efficiently put the values in the right position of the matrix.

  Yes, but where are you computing the matrix entries? All on the first process? On all the processes? Or on some random process? In order to perform efficiently the code needs to generate the vast majority of the matrix entries ON the process where they will eventually be stored. This means the code needs to be aware of the (good) data decomposition that allows this. The DMDA object is one way to manage this decomposition (that is easy to use), of course you can mange the decomposition yourself but it still needs to be managed no matter what. Actually creating the matrix entries is not the hard part, the hard part is setting up the (good) decomposition and writing the code that uses that decomposition.

  Barry

> 
> 
> Regards,
> Maahi Talukder
> 
> On Tue, Feb 12, 2019 at 9:03 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   Sounds like you have a single structured grid so you should use https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDACreate2d.html and start with something like src/ksp/ksp/examples/tutorials/ex29.c and ex46.c 
> 
>   The DMDA manages dividing up the grid among processes and simplifies generating the matrix in parallel.
> 
>   Good luck,
> 
>    Barry
> 
> 
> > On Feb 12, 2019, at 7:43 PM, Maahi Talukder <maahi.buet at gmail.com> wrote:
> > 
> > I am using Finite Difference to discretize my system and I am working with nine point stencil. So I am populating in Q() all the values. The code to do that is the following - 
> > 
> > Do i = 2,ymax-1
> > 
> > Do j = 2,xmax-1
> > 
> > g11(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i,j+1)-x(i,j-1)) +0.25*(y(i,j+1)-y(i,j-1))*(y(i,j+1)-y(i,j-1))
> > g12(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i+1,j)-x(i-1,j))+ 0.25*(y(i,j+1)-y(i,j-1))*(y(i+1,j)-y(i-1,j))
> > g22(i-1,j-1) = 0.25*(x(i+1,j)-x(i-1,j))*(x(i+1,j)-x(i-1,j)) + 0.25*(y(i+1,j)-y(i-1,j))*(y(i+1,j)-y(i-1,j))
> > 
> > M(j-1,j-1+xmax*(i-2))= (-1)*(g12(i-1,j-1)/2)
> > M(j-1,j+xmax*(i-2))= g11(i-1,j-1)
> > M(j-1,j+1+xmax*(i-2)) = g12(i-1,j-1)/2
> > M(j-1,j-1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> > M(j-1,j+xmax*(i-2)+xmax)= (-2)*(g22(i-1,j-1)+g11(i-1,j-1))
> > M(j-1,j+1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> > M(j-1,j-1+xmax*(i-2)+2*xmax) = g12(i-1,j-1)/2
> > M(j-1,j+xmax*(i-2)+2*xmax) = g11(i-1,j-1)
> > M(j-1,j+1+xmax*(i-2)+2*xmax) = (-1)*(g12(i-1,j-1)/2)       
> > 
> > end Do
> > 
> > N(1+(xmax-2)*(i-2):(xmax-2)*(i-1),1:xmax*ymax)= M(1:xmax-2,1:xmax*ymax)
> > M = 0
> > 
> > end Do
> > 
> > E(1:(xmax-2)*(ymax-2),1:xmax*ymax-2*xmax) = N(1:(xmax-2)*(ymax-2),xmax+1:xmax*ymax-xmax)
> > 
> > Do i = 1,ymax-2
> >     
> > Q(1:(xmax-2)*(ymax-2),1+(xmax-2)*(i-1):(xmax-2)*i)= E(1:(xmax-2)*(ymax-2),2+xmax*(i-1):(xmax-1)+xmax*(i-1))
> >     
> > end Do
> > 
> > So how do I go about decomposing this Q() matrix across processes ? What PETSc function do I use for that?
> > 
> > 
> > Regards,
> > Maahi
> > 
> > On Tue, Feb 12, 2019 at 8:25 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> > 
> >   With MPI parallelism you need to do a decomposition of your data across the processes. Thus, for example, each process will generate a subset of the matrix entries. In addition, for large problems (which is what parallelism is for) you cannot use "dense" data structures, like your Q(), to store sparse matrices. For large problems with true sparse matrices the Q() will not fit on your machine and consists almost completely of zeros which it makes no sense to save in memory.
> > 
> >    How are your Q() entries being generated? This will determine the decomposition of the data you need to make. For example, with the finite element method each process would be assigned a subset of the elements (generally for a subregion of the entire domain).
> > 
> >    Barry
> > 
> > 
> > 
> > > On Feb 12, 2019, at 5:42 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > > 
> > > 
> > > Dear All,
> > > 
> > > 
> > > I am tying to solve a linear system using ksp solvers. I have managed to solve the system with a sequential code. The part of my sequential code that deals with creating Matrix and setting values is as the following - 
> > > 
> > > call MatCreate(PETSC_COMM_WORLD,Mp,ierr)
> > > call MatSetSizes(Mp,PETSC_DECIDE,PETSC_DECIDE,u*v,u*v,ierr)
> > > call MatSetFromOptions(Mp,ierr)
> > > call MatSetUp(Mp,ierr)
> > > 
> > > Do p = 1,29008
> > > Do r = 1,29008
> > > if(Q(p,r)/=0.0) then
> > > val(1) = Q(p,r)
> > > col(1) = r-1
> > > call MatSetValues(Mp,ione,p-1,ione,col,val,INSERT_VALUES,ierr)
> > > endif
> > > end Do
> > > end Do
> > > 
> > > call MatAssemblyBegin(Mp,MAT_FINAL_ASSEMBLY,ierr)
> > > call MatAssemblyEnd(Mp,MAT_FINAL_ASSEMBLY,ierr)
> > > 
> > > And the part of my sequential code that creates the vector is -
> > > 
> > > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,u*v,Bx,ierr)
> > > call VecSetFromOptions(Bx,ierr)
> > > call VecDuplicate(Bx,Xp,ierr)
> > > call VecSet(Bx,zero,ierr) 
> > > 
> > > Do p = 1,29008
> > > if(Fx(p,1)/=0.0) then
> > > val(1) = Fx(p,1)
> > > call VecSetValues(Bx,ione,p-1,val,INSERT_VALUES,ierr)
> > > endif
> > > end Do
> > > 
> > > call VecAssemblyBegin(Bx,ierr)
> > > call VecAssemblyEnd(Bx,ierr)
> > > 
> > > So when I run the code on single processor, it runs fine. But when I tried to run it on more than one processor, it failed.  Now from what I understood from going through the manual is that if I use MatCreate to create my Matrix, then depending on the no of processor  that I put in after 'mpiexec -n ...' , it either acts either as a sequential code or a parallel code. And I don't need to anything extra to make it work in parallel, as PETSc does that internally. 
> > > 
> > > So would you please let me know  what modifications I need to do to my existing sequential code to make it work in parallel like using MatGetOwnershipRange ? 
> > > 
> > > Regards,
> > > Maahi Talukder
> > > MSc Student
> > > Clarkson University
> > > 
> > > 
> > > 
> > 
> 


From KJiao at slb.com  Wed Feb 13 06:25:45 2019
From: KJiao at slb.com (Kun Jiao)
Date: Wed, 13 Feb 2019 12:25:45 +0000
Subject: [petsc-users] interchange between two different DMDAs
Message-ID: <VI1PR0402MB353555837478EAA7B0FA7BAAAE660@VI1PR0402MB3535.eurprd04.prod.outlook.com>

Hi,

I have a problem that need to interchange between two different DMDAs. They have the exactly same size, however they are distributed/partitioned differently across multiple processes. For instance, one DMDA is created with DMDACreated1d, the other is created with DMDACreated3d. How can I interchange between different DMDAs with VecScatter, or something else?

Regards,
Kun


Schlumberger-Private
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190213/54d073f8/attachment-0001.html>

From knepley at gmail.com  Wed Feb 13 06:55:28 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 13 Feb 2019 07:55:28 -0500
Subject: [petsc-users] interchange between two different DMDAs
In-Reply-To: <VI1PR0402MB353555837478EAA7B0FA7BAAAE660@VI1PR0402MB3535.eurprd04.prod.outlook.com>
References: <VI1PR0402MB353555837478EAA7B0FA7BAAAE660@VI1PR0402MB3535.eurprd04.prod.outlook.com>
Message-ID: <CAMYG4GnkB06Rw-u8VVrNEVZp32osBbi0__=0gpaVWM9B-Yj9fw@mail.gmail.com>

On Wed, Feb 13, 2019 at 7:25 AM Kun Jiao via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
>
>
> I have a problem that need to interchange between two different DMDAs.
> They have the exactly same size, however they are distributed/partitioned
> differently across multiple processes. For instance, one DMDA is created
> with DMDACreated1d, the other is created with DMDACreated3d. How can I
> interchange between different DMDAs with VecScatter, or something else?
>

Yes, you would need to use VecScatter. Getting this right sounds
nontrivial. What are you using it for?

  Thanks,

     Matt


>
>
> Regards,
>
> Kun
>
> Schlumberger-Private
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190213/affddc7c/attachment.html>

From dave.mayhem23 at gmail.com  Wed Feb 13 10:38:54 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Wed, 13 Feb 2019 16:38:54 +0000
Subject: [petsc-users] interchange between two different DMDAs
In-Reply-To: <CAMYG4GnkB06Rw-u8VVrNEVZp32osBbi0__=0gpaVWM9B-Yj9fw@mail.gmail.com>
References: <VI1PR0402MB353555837478EAA7B0FA7BAAAE660@VI1PR0402MB3535.eurprd04.prod.outlook.com>
	<CAMYG4GnkB06Rw-u8VVrNEVZp32osBbi0__=0gpaVWM9B-Yj9fw@mail.gmail.com>
Message-ID: <CAJ98EDoqSsTCympB474jr-RqP05mg7eD3bsHZaeOAjWW4TSwOQ@mail.gmail.com>

On Wed, 13 Feb 2019 at 12:56, Matthew Knepley via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> On Wed, Feb 13, 2019 at 7:25 AM Kun Jiao via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>>
>>
>> I have a problem that need to interchange between two different DMDAs.
>> They have the exactly same size, however they are distributed/partitioned
>> differently across multiple processes. For instance, one DMDA is created
>> with DMDACreated1d, the other is created with DMDACreated3d. How can I
>> interchange between different DMDAs with VecScatter, or something else?
>>
>
> Yes, you would need to use VecScatter. Getting this right sounds
> nontrivial. What are you using it for?
>

Kun - we actually do something similar to what you want internal to
PCTELESCOPE when repartitioning the DMDAs.
See here

https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/telescope/telescope_dmda.c.html

The big difference in PCTELESCOPE is that we building scatters between 2
DMDAs with the same spatial dimension, and different parallel partitions.
That said, I don't think moving data between a 1D and 3D DMDA would
complicate things greatly, but you won't find any examples within
PCTELESCOPE.

Thanks.
Dave



>
>   Thanks,
>
>      Matt
>
>
>>
>>
>> Regards,
>>
>> Kun
>>
>> Schlumberger-Private
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190213/b2203262/attachment.html>

From mbuerkle at web.de  Thu Feb 14 18:44:59 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Fri, 15 Feb 2019 01:44:59 +0100
Subject: [petsc-users] superlu_dist+parmetis
Message-ID: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>

Dear PETSc team,

I try to run superlu_dist+parmetis with " -mat_superlu_dist_colperm parmetis -mat_superlu_dist_parsymbfact" which gives me the following error

[6]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[651]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[653]PETSC ERROR: ------------------------------------------------------------------------
[653]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[653]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[653]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[653]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[653]PETSC ERROR: likely location of problem given in stack below
[653]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[653]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[653]PETSC ERROR:       INSTEAD the line number of the start of the function
[653]PETSC ERROR:       is given.
[653]PETSC ERROR: [653] SuperLU_DIST:pzgssvx line 465 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[653]PETSC ERROR: [653] MatLUFactorNumeric_SuperLU_DIST line 314 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[653]PETSC ERROR: [653] MatLUFactorNumeric line 3124 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
[653]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[653]PETSC ERROR: Signal received
[653]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[653]PETSC ERROR: Petsc Development GIT revision: v3.10.3-980-g66b342c  GIT Date: 2018-12-26 13:49:21 -0600
[653]PETSC ERROR: /home/cdfmat_marius/prog/transomat_latest_openmpi4.0/transomat on a  named h023 by cdfmat_marius Wed Feb 13 23:58:21 2019

Any idea?

best,
marius

From rlmackie862 at gmail.com  Thu Feb 14 19:33:30 2019
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 14 Feb 2019 17:33:30 -0800
Subject: [petsc-users] GPUs, cud, complex
Message-ID: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>

We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
We have installed PETSc following the instructions here:
https://www.mcs.anl.gov/petsc/features/gpus.html <https://www.mcs.anl.gov/petsc/features/gpus.html>

Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.

Is it possible to run PETSc on GPUS in complex mode?d


Thanks, Randy M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190214/1b9c255e/attachment.html>

From bsmith at mcs.anl.gov  Thu Feb 14 21:16:09 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 15 Feb 2019 03:16:09 +0000
Subject: [petsc-users] superlu_dist+parmetis
In-Reply-To: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
References: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
Message-ID: <55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>


  Given the message the likely cause is SuperLU_DIST got a zero pivot on process 6. Presumably the colperm parmetis is one induced the zero pivot.

  Have you tried other superlu_dist options to find others that do not cause this?

   Barry


> On Feb 14, 2019, at 6:44 PM, Marius Buerkle via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Dear PETSc team,
> 
> I try to run superlu_dist+parmetis with " -mat_superlu_dist_colperm parmetis -mat_superlu_dist_parsymbfact" which gives me the following error
> 
> [6]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [651]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [653]PETSC ERROR: ------------------------------------------------------------------------
> [653]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [653]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [653]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [653]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [653]PETSC ERROR: likely location of problem given in stack below
> [653]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [653]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [653]PETSC ERROR:       INSTEAD the line number of the start of the function
> [653]PETSC ERROR:       is given.
> [653]PETSC ERROR: [653] SuperLU_DIST:pzgssvx line 465 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [653]PETSC ERROR: [653] MatLUFactorNumeric_SuperLU_DIST line 314 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [653]PETSC ERROR: [653] MatLUFactorNumeric line 3124 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> [653]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [653]PETSC ERROR: Signal received
> [653]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [653]PETSC ERROR: Petsc Development GIT revision: v3.10.3-980-g66b342c  GIT Date: 2018-12-26 13:49:21 -0600
> [653]PETSC ERROR: /home/cdfmat_marius/prog/transomat_latest_openmpi4.0/transomat on a  named h023 by cdfmat_marius Wed Feb 13 23:58:21 2019
> 
> Any idea?
> 
> best,
> marius


From mbuerkle at web.de  Fri Feb 15 00:30:11 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Fri, 15 Feb 2019 07:30:11 +0100
Subject: [petsc-users] superlu_dist+parmetis
In-Reply-To: <55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>
References: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
	<55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>
Message-ID: <trinity-72ef7874-70fe-4eb1-a167-757b11df22b0-1550212211771@3c-app-webde-bs19>

It works with all options for "-mat_superlu_dist_colperm" save parmetis. or do you mean to change some other option besides colperm?

> Gesendet: Freitag, 15. Februar 2019 um 04:16 Uhr
> Von: "Smith, Barry F." <bsmith at mcs.anl.gov>
> An: "Marius Buerkle" <mbuerkle at web.de>
> Cc: "PETSc users list" <petsc-users at mcs.anl.gov>
> Betreff: Re: [petsc-users] superlu_dist+parmetis
>
>
>   Given the message the likely cause is SuperLU_DIST got a zero pivot on process 6. Presumably the colperm parmetis is one induced the zero pivot.
>
>   Have you tried other superlu_dist options to find others that do not cause this?
>
>    Barry
>
>
> > On Feb 14, 2019, at 6:44 PM, Marius Buerkle via petsc-users <petsc-users at mcs.anl.gov> wrote:
> >
> > Dear PETSc team,
> >
> > I try to run superlu_dist+parmetis with " -mat_superlu_dist_colperm parmetis -mat_superlu_dist_parsymbfact" which gives me the following error
> >
> > [6]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [651]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> > [653]PETSC ERROR: ------------------------------------------------------------------------
> > [653]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> > [653]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [653]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [653]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [653]PETSC ERROR: likely location of problem given in stack below
> > [653]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> > [653]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> > [653]PETSC ERROR:       INSTEAD the line number of the start of the function
> > [653]PETSC ERROR:       is given.
> > [653]PETSC ERROR: [653] SuperLU_DIST:pzgssvx line 465 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> > [653]PETSC ERROR: [653] MatLUFactorNumeric_SuperLU_DIST line 314 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> > [653]PETSC ERROR: [653] MatLUFactorNumeric line 3124 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> > [653]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [653]PETSC ERROR: Signal received
> > [653]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [653]PETSC ERROR: Petsc Development GIT revision: v3.10.3-980-g66b342c  GIT Date: 2018-12-26 13:49:21 -0600
> > [653]PETSC ERROR: /home/cdfmat_marius/prog/transomat_latest_openmpi4.0/transomat on a  named h023 by cdfmat_marius Wed Feb 13 23:58:21 2019
> >
> > Any idea?
> >
> > best,
> > marius
>
>

From bsmith at mcs.anl.gov  Fri Feb 15 00:38:39 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 15 Feb 2019 06:38:39 +0000
Subject: [petsc-users] superlu_dist+parmetis
In-Reply-To: <trinity-72ef7874-70fe-4eb1-a167-757b11df22b0-1550212211771@3c-app-webde-bs19>
References: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
	<55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>
	<trinity-72ef7874-70fe-4eb1-a167-757b11df22b0-1550212211771@3c-app-webde-bs19>
Message-ID: <5A25FA95-0301-470D-8B4A-813ADFCB9C72@mcs.anl.gov>



> On Feb 15, 2019, at 12:30 AM, Marius Buerkle <mbuerkle at web.de> wrote:
> 
> It works with all options for "-mat_superlu_dist_colperm" save parmetis. or do you mean to change some other option besides colperm?

  No, I didn't have any other suggestions. It could be that this option just more easily introduces zero pivots or you could report it to Sherry Li, the main SuperLU_DIST developer and see what she says.

   Barry

> 
>> Gesendet: Freitag, 15. Februar 2019 um 04:16 Uhr
>> Von: "Smith, Barry F." <bsmith at mcs.anl.gov>
>> An: "Marius Buerkle" <mbuerkle at web.de>
>> Cc: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Betreff: Re: [petsc-users] superlu_dist+parmetis
>> 
>> 
>>  Given the message the likely cause is SuperLU_DIST got a zero pivot on process 6. Presumably the colperm parmetis is one induced the zero pivot.
>> 
>>  Have you tried other superlu_dist options to find others that do not cause this?
>> 
>>   Barry
>> 
>> 
>>> On Feb 14, 2019, at 6:44 PM, Marius Buerkle via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> Dear PETSc team,
>>> 
>>> I try to run superlu_dist+parmetis with " -mat_superlu_dist_colperm parmetis -mat_superlu_dist_parsymbfact" which gives me the following error
>>> 
>>> [6]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> [651]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>> [653]PETSC ERROR: ------------------------------------------------------------------------
>>> [653]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
>>> [653]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [653]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [653]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> [653]PETSC ERROR: likely location of problem given in stack below
>>> [653]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>> [653]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> [653]PETSC ERROR:       INSTEAD the line number of the start of the function
>>> [653]PETSC ERROR:       is given.
>>> [653]PETSC ERROR: [653] SuperLU_DIST:pzgssvx line 465 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>>> [653]PETSC ERROR: [653] MatLUFactorNumeric_SuperLU_DIST line 314 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>>> [653]PETSC ERROR: [653] MatLUFactorNumeric line 3124 /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
>>> [653]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [653]PETSC ERROR: Signal received
>>> [653]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [653]PETSC ERROR: Petsc Development GIT revision: v3.10.3-980-g66b342c  GIT Date: 2018-12-26 13:49:21 -0600
>>> [653]PETSC ERROR: /home/cdfmat_marius/prog/transomat_latest_openmpi4.0/transomat on a  named h023 by cdfmat_marius Wed Feb 13 23:58:21 2019
>>> 
>>> Any idea?
>>> 
>>> best,
>>> marius
>> 
>> 


From M.Tiberga at tudelft.nl  Fri Feb 15 05:26:26 2019
From: M.Tiberga at tudelft.nl (Marco Tiberga)
Date: Fri, 15 Feb 2019 11:26:26 +0000
Subject: [petsc-users] Missing Fortran interface for PetscFree?
Message-ID: <20f81576c3c0450fbafd4fec364f9130@tudelft.nl>

Dear PETSc developers,

Unfortunately, I need to continue the previous conversation (which I attached), because I am getting another compiler error.
I need to call PetscFree() after PCFieldSplitGetSubKSP (I am trying to implement what is described in lines 117?120 of this example<https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html>), but the compiler complains that the reference to it is undefined, even if I ?use petscsys?.

So, I was wondering whether the Fortran interface for PetscFree is missing as well, or if I am doing something wrong.

Thanks again for the help!

Best regards,

Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/



From: Marco Tiberga
Sent: donderdag 7 februari 2019 15:31
To: 'Smith, Barry F.'
Cc: petsc-users at mcs.anl.gov; Danny Lathouwers - TNW
Subject: RE: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS and PCFieldSplitGetIS?

Dear Barry,

Thanks a lot for your fast reply, as usual.
The patch did work!

With kind regards,
Marco Tiberga
PhD candidate
Delft University of Technology
Faculty of Applied Sciences
Radiation Science & Technology Department
Mekelweg 15, 2629 JB Delft, The Netherlands
E-Mail: m.tiberga at tudelft.nl<mailto:m.tiberga at tudelft.nl>
Website: http://www.nera.rst.tudelft.nl/



From: Smith, Barry F. [mailto:bsmith at mcs.anl.gov]
Sent: donderdag 7 februari 2019 5:19
To: Marco Tiberga
Cc: petsc-users at mcs.anl.gov; Danny Lathouwers - TNW
Subject: Re: [petsc-users] Missing Fortran interface for PCFieldSplitSetIS and PCFieldSplitGetIS?


   Marco,
   We were missing a Fortran stub in the code. You can either apply the attached patch or use the git branch barry/add-pcfieldsplit-getis/maint for the missing functionality. The fix will also be in the next maintenance release of petsc-3.10

   Please let us know if there are any difficulties.

   Barry


> On Feb 6, 2019, at 9:56 AM, Marco Tiberga via petsc-users <petsc-users at mcs.anl.gov> wrote:
>
> Dear PETSc developers,
>
> I am developing a coupled solver for the incompressible Navier-Stokes equations, based on the PCFIELDSPLIT implemented in PETSc (I?m using version 3.10.1).
> My code is written in Fortran.
>
> I tried to follow as much as possible this example, but I am getting weird errors. I think the problem is in the call to PCFieldSplitSetIS.
> To debug the code, for each field I wanted to call PCFieldSplitGetIS (after setting it) and print the IS with ISView. However, the compiler says the reference to PCFieldSplitGetIS is undefined.
>
> So, I was wondering whether PCFieldSplitGetIS and PCFieldSplitSetIS are actually available for Fortran.
> I tried to had a look at petsc-3.10.1/src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90, but I cannot find the interfaces for these functions.
>
> Thanks for the clarification.
>
> Best regards,
>
> Marco Tiberga
> PhD candidate
> Delft University of Technology
> Faculty of Applied Sciences
> Radiation Science & Technology Department
> Mekelweg 15, 2629 JB Delft, The Netherlands
> E-Mail: m.tiberga at tudelft.nl
> Website: http://www.nera.rst.tudelft.nl/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/5d00d710/attachment-0001.html>

From knepley at gmail.com  Fri Feb 15 08:55:38 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 15 Feb 2019 09:55:38 -0500
Subject: [petsc-users] Missing Fortran interface for PetscFree?
In-Reply-To: <20f81576c3c0450fbafd4fec364f9130@tudelft.nl>
References: <20f81576c3c0450fbafd4fec364f9130@tudelft.nl>
Message-ID: <CAMYG4Gn6mPVNdjnVfTWxpZwNWyMza4TA2KEW_rykcBgD5C2e3w@mail.gmail.com>

On Fri, Feb 15, 2019 at 6:26 AM Marco Tiberga via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear PETSc developers,
>
>
>
> Unfortunately, I need to continue the previous conversation (which I
> attached), because I am getting another compiler error.
>
> I need to call PetscFree() after PCFieldSplitGetSubKSP (I am trying to
> implement what is described in lines 117?120 of this example
> <https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html>),
> but the compiler complains that the reference to it is undefined, even if I
> ?use petscsys?.
>
>
>
> So, I was wondering whether the Fortran interface for PetscFree is missing
> as well, or if I am doing something wrong.
>

PetscMalloc() and PetscFree() cannot work for Fortran because F90 has its
own idea of pointers.

However, the Fortran binding for PCFieldSplitGetSubKSP() does not allocate
the array. Rather, you pass in an array
of the correct size and it just fills it up.

  Thanks,

     Matt

>
>
> Thanks again for the help!
>
>
>
> Best regards,
>
>
>
> Marco Tiberga
>
> PhD candidate
>
> Delft University of Technology
>
> Faculty of Applied Sciences
>
> Radiation Science & Technology Department
>
> Mekelweg 15, 2629 JB Delft, The Netherlands
>
> E-Mail: *m.tiberga at tudelft.nl <m.tiberga at tudelft.nl>*
>
> Website: http://www.nera.rst.tudelft.nl/
>
>
>
>
>
>
>
> *From:* Marco Tiberga
> *Sent:* donderdag 7 februari 2019 15:31
> *To:* 'Smith, Barry F.'
> *Cc:* petsc-users at mcs.anl.gov; Danny Lathouwers - TNW
> *Subject:* RE: [petsc-users] Missing Fortran interface for
> PCFieldSplitSetIS and PCFieldSplitGetIS?
>
>
>
> Dear Barry,
>
>
>
> Thanks a lot for your fast reply, as usual.
>
> The patch did work!
>
>
>
> With kind regards,
>
> Marco Tiberga
>
> PhD candidate
>
> Delft University of Technology
>
> Faculty of Applied Sciences
>
> Radiation Science & Technology Department
>
> Mekelweg 15, 2629 JB Delft, The Netherlands
>
> E-Mail: *m.tiberga at tudelft.nl <m.tiberga at tudelft.nl>*
>
> Website: http://www.nera.rst.tudelft.nl/
>
>
>
>
>
>
>
> *From:* Smith, Barry F. [mailto:bsmith at mcs.anl.gov]
> *Sent:* donderdag 7 februari 2019 5:19
> *To:* Marco Tiberga
> *Cc:* petsc-users at mcs.anl.gov; Danny Lathouwers - TNW
> *Subject:* Re: [petsc-users] Missing Fortran interface for
> PCFieldSplitSetIS and PCFieldSplitGetIS?
>
>
>
>
>    Marco,
>
>    We were missing a Fortran stub in the code. You can either apply the
> attached patch or use the git branch barry/add-pcfieldsplit-getis/maint for
> the missing functionality. The fix will also be in the next maintenance
> release of petsc-3.10
>
>    Please let us know if there are any difficulties.
>
>    Barry
>
>
> > On Feb 6, 2019, at 9:56 AM, Marco Tiberga via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Dear PETSc developers,
> >
> > I am developing a coupled solver for the incompressible Navier-Stokes
> equations, based on the PCFIELDSPLIT implemented in PETSc (I?m using
> version 3.10.1).
> > My code is written in Fortran.
> >
> > I tried to follow as much as possible this example, but I am getting
> weird errors. I think the problem is in the call to PCFieldSplitSetIS.
> > To debug the code, for each field I wanted to call PCFieldSplitGetIS
> (after setting it) and print the IS with ISView. However, the compiler says
> the reference to PCFieldSplitGetIS is undefined.
> >
> > So, I was wondering whether PCFieldSplitGetIS and PCFieldSplitSetIS are
> actually available for Fortran.
> > I tried to had a look at
> petsc-3.10.1/src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90, but I cannot
> find the interfaces for these functions.
> >
> > Thanks for the clarification.
> >
> > Best regards,
> >
> > Marco Tiberga
> > PhD candidate
> > Delft University of Technology
> > Faculty of Applied Sciences
> > Radiation Science & Technology Department
> > Mekelweg 15, 2629 JB Delft, The Netherlands
> > E-Mail: m.tiberga at tudelft.nl
> > Website: http://www.nera.rst.tudelft.nl/
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/d014cadf/attachment.html>

From xsli at lbl.gov  Fri Feb 15 09:01:19 2019
From: xsli at lbl.gov (Xiaoye S. Li)
Date: Fri, 15 Feb 2019 07:01:19 -0800
Subject: [petsc-users] superlu_dist+parmetis
In-Reply-To: <5A25FA95-0301-470D-8B4A-813ADFCB9C72@mcs.anl.gov>
References: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
	<55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>
	<trinity-72ef7874-70fe-4eb1-a167-757b11df22b0-1550212211771@3c-app-webde-bs19>
	<5A25FA95-0301-470D-8B4A-813ADFCB9C72@mcs.anl.gov>
Message-ID: <CAFvbobWGfhkuuhPSXJT0G+uBU_RxbuJYgxwPn1Y-49Q3bO9Ozg@mail.gmail.com>

I am pretty sure this is a bug in parmetis.  A few years ago, I nailed down
this "divide by zero" bug, and reported to them. They said they would take
a look, but never did.  This usually happens when the graph is relatively
dense.

You can try sequential Metis.

Sherry

On Thu, Feb 14, 2019 at 10:38 PM Smith, Barry F. via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
>
> > On Feb 15, 2019, at 12:30 AM, Marius Buerkle <mbuerkle at web.de> wrote:
> >
> > It works with all options for "-mat_superlu_dist_colperm" save parmetis.
> or do you mean to change some other option besides colperm?
>
>   No, I didn't have any other suggestions. It could be that this option
> just more easily introduces zero pivots or you could report it to Sherry
> Li, the main SuperLU_DIST developer and see what she says.
>
>    Barry
>
> >
> >> Gesendet: Freitag, 15. Februar 2019 um 04:16 Uhr
> >> Von: "Smith, Barry F." <bsmith at mcs.anl.gov>
> >> An: "Marius Buerkle" <mbuerkle at web.de>
> >> Cc: "PETSc users list" <petsc-users at mcs.anl.gov>
> >> Betreff: Re: [petsc-users] superlu_dist+parmetis
> >>
> >>
> >>  Given the message the likely cause is SuperLU_DIST got a zero pivot on
> process 6. Presumably the colperm parmetis is one induced the zero pivot.
> >>
> >>  Have you tried other superlu_dist options to find others that do not
> cause this?
> >>
> >>   Barry
> >>
> >>
> >>> On Feb 14, 2019, at 6:44 PM, Marius Buerkle via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >>>
> >>> Dear PETSc team,
> >>>
> >>> I try to run superlu_dist+parmetis with " -mat_superlu_dist_colperm
> parmetis -mat_superlu_dist_parsymbfact" which gives me the following error
> >>>
> >>> [6]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> Exception,probably divide by zero
> >>> [6]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >>> [6]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> >>> [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
> OS X to find memory corruption errors
> >>> [651]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> >>> [653]PETSC ERROR:
> ------------------------------------------------------------------------
> >>> [653]PETSC ERROR: Caught signal number 15 Terminate: Some process (or
> the batch system) has told this process to end
> >>> [653]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >>> [653]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> >>> [653]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
> Mac OS X to find memory corruption errors
> >>> [653]PETSC ERROR: likely location of problem given in stack below
> >>> [653]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> >>> [653]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> >>> [653]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> >>> [653]PETSC ERROR:       is given.
> >>> [653]PETSC ERROR: [653] SuperLU_DIST:pzgssvx line 465
> /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> >>> [653]PETSC ERROR: [653] MatLUFactorNumeric_SuperLU_DIST line 314
> /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> >>> [653]PETSC ERROR: [653] MatLUFactorNumeric line 3124
> /home/cdfmat_marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> >>> [653]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> >>> [653]PETSC ERROR: Signal received
> >>> [653]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> >>> [653]PETSC ERROR: Petsc Development GIT revision:
> v3.10.3-980-g66b342c  GIT Date: 2018-12-26 13:49:21 -0600
> >>> [653]PETSC ERROR:
> /home/cdfmat_marius/prog/transomat_latest_openmpi4.0/transomat on a  named
> h023 by cdfmat_marius Wed Feb 13 23:58:21 2019
> >>>
> >>> Any idea?
> >>>
> >>> best,
> >>> marius
> >>
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/f21dfb89/attachment.html>

From maahi.buet at gmail.com  Fri Feb 15 13:22:53 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Fri, 15 Feb 2019 14:22:53 -0500
Subject: [petsc-users] KSP ex2.c and Parallelization
Message-ID: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>

Hello All,

I have some questions regarding ex2.c in /
*petsc/src/ksp/ksp/examples/tutorials/ex2.c. *

1.   The  local sizes of the matrix was decided by PETSc using *ierr =
MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);*
But without knowing the local sizes, how was the following function used
which requires the knowledge of how the big matrix was split into smaller
chunks within PETSc?
  *ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);*

2.  When using  *MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr),  *the
global index of  first and last row of which local chunk of the big matrix
is it returning ? I mean if the big matrix was split into three processes,
for example, fist and last global row of which process is it returning?

Your replies are highly appreciated

Thanks,
Maahi Talukder
Clarkson University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/11942dd0/attachment.html>

From hzhang at mcs.anl.gov  Fri Feb 15 13:43:06 2019
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Fri, 15 Feb 2019 19:43:06 +0000
Subject: [petsc-users] KSP ex2.c and Parallelization
In-Reply-To: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>
References: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>
Message-ID: <CAGCphBvrgJd+reQRydVuGzjr6KUbdh8vUbCcJqAm67kovvPNtw@mail.gmail.com>

Maahi :
Hello All,

I have some questions regarding ex2.c in /petsc/src/ksp/ksp/examples/tutorials/ex2.c.

1.   The  local sizes of the matrix was decided by PETSc using ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
But without knowing the local sizes, how was the following function used which requires the knowledge of how the big matrix was split into smaller chunks within PETSc?
  ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);
This example sets a matrix for 2D 5-point PDE stencil, for which the maximum non-zeros in each row is 5.

2.  When using  MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr),  the global index of  first and last row of which local chunk of the big matrix is it returning ? I mean if the big matrix was split into three processes, for example, fist and last global row of which process is it returning?
Adding a line
  ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
+  printf("Istart %d Iend %d\n",Istart,Iend);

I get
$ mpiexec -n 3 ./ex2
Istart 0 Iend 19
Istart 19 Iend 38
Istart 38 Iend 56
Norm of error 0.000522061 iterations 9
The matrix global size = 56

Hong

Your replies are highly appreciated

Thanks,
Maahi Talukder
Clarkson University


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/c98003e6/attachment.html>

From maahi.buet at gmail.com  Fri Feb 15 16:26:09 2019
From: maahi.buet at gmail.com (Maahi Talukder)
Date: Fri, 15 Feb 2019 17:26:09 -0500
Subject: [petsc-users] KSP ex2.c and Parallelization
In-Reply-To: <CAGCphBvrgJd+reQRydVuGzjr6KUbdh8vUbCcJqAm67kovvPNtw@mail.gmail.com>
References: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>
	<CAGCphBvrgJd+reQRydVuGzjr6KUbdh8vUbCcJqAm67kovvPNtw@mail.gmail.com>
Message-ID: <CAJOaupEeSBDQ05AAE-c=9AamL9AWr_BWRCP5r2y5nkZMHoD8LQ@mail.gmail.com>

Thank you so much for your reply.

But in the definition of d_nz of *MatMPIAIJSetPreallocation  *, it is
defined as 'number of nonzeros per row in DIAGONAL portion of local
submatrix'. So I was wondering about how to find the Diagonal portion of
the local submatrix that PETSc is splitting the big matrix into . So should
it be the maximum number of nonzeros in each row of the big matrix instead
of  the min number of nonzeros in the diagonal portion of the local matrix?

And so MatGetOwnershipRange returns the start and the end of all the
smaller matrices that PETSc is splitting the big matrix into, right?


Regards,
Maahi Talukder

On Fri, Feb 15, 2019 at 2:43 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Maahi :
>
>> Hello All,
>>
>> I have some questions regarding ex2.c in /
>> *petsc/src/ksp/ksp/examples/tutorials/ex2.c. *
>>
>> 1.   The  local sizes of the matrix was decided by PETSc using *ierr =
>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);*
>> But without knowing the local sizes, how was the following function used
>> which requires the knowledge of how the big matrix was split into smaller
>> chunks within PETSc?
>>   *ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);*
>>
> This example sets a matrix for 2D 5-point PDE stencil, for which the
> maximum non-zeros in each row is 5.
>
>>
>> 2.  When using  *MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr),  *the
>> global index of  first and last row of which local chunk of the big matrix
>> is it returning ? I mean if the big matrix was split into three processes,
>> for example, fist and last global row of which process is it returning?
>>
> Adding a line
>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> +  printf("Istart %d Iend %d\n",Istart,Iend);
>
> I get
> $ mpiexec -n 3 ./ex2
> Istart 0 Iend 19
> Istart 19 Iend 38
> Istart 38 Iend 56
> Norm of error 0.000522061 iterations 9
> The matrix global size = 56
>
> Hong
>
>>
>> Your replies are highly appreciated
>>
>> Thanks,
>> Maahi Talukder
>> Clarkson University
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190215/3e070761/attachment.html>

From bsmith at mcs.anl.gov  Fri Feb 15 16:47:23 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 15 Feb 2019 22:47:23 +0000
Subject: [petsc-users] KSP ex2.c and Parallelization
In-Reply-To: <CAJOaupEeSBDQ05AAE-c=9AamL9AWr_BWRCP5r2y5nkZMHoD8LQ@mail.gmail.com>
References: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>
	<CAGCphBvrgJd+reQRydVuGzjr6KUbdh8vUbCcJqAm67kovvPNtw@mail.gmail.com>
	<CAJOaupEeSBDQ05AAE-c=9AamL9AWr_BWRCP5r2y5nkZMHoD8LQ@mail.gmail.com>
Message-ID: <8235A67A-7378-4A79-B5C9-8B48802D57AE@anl.gov>


   Maybe we could provide more guidance on how to proceed if we knew a bit more about the 1) problem you are trying to solve, 2) the discretization you are using and 3) the mesh you are using?

    From your code fragment with a m*n I am guessing you are solving something on a structured grid with m grid points in the x direction and n grid points in the y direction? 

  Barry


> On Feb 15, 2019, at 4:26 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thank you so much for your reply. 
>  
> But in the definition of d_nz of MatMPIAIJSetPreallocation  , it is defined as 'number of nonzeros per row in DIAGONAL portion of local submatrix'. So I was wondering about how to find the Diagonal portion of the local submatrix that PETSc is splitting the big matrix into . So should it be the maximum number of nonzeros in each row of the big matrix instead of  the min number of nonzeros in the diagonal portion of the local matrix?
> 
> And so MatGetOwnershipRange returns the start and the end of all the smaller matrices that PETSc is splitting the big matrix into, right?
> 
> 
> Regards,
> Maahi Talukder
> 
> On Fri, Feb 15, 2019 at 2:43 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> Maahi :
> Hello All,
> 
> I have some questions regarding ex2.c in /petsc/src/ksp/ksp/examples/tutorials/ex2.c. 
> 
> 1.   The  local sizes of the matrix was decided by PETSc using ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
> But without knowing the local sizes, how was the following function used which requires the knowledge of how the big matrix was split into smaller chunks within PETSc? 
>   ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);
> This example sets a matrix for 2D 5-point PDE stencil, for which the maximum non-zeros in each row is 5. 
> 
> 2.  When using  MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr),  the global index of  first and last row of which local chunk of the big matrix is it returning ? I mean if the big matrix was split into three processes, for example, fist and last global row of which process is it returning? 
> Adding a line 
>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> +  printf("Istart %d Iend %d\n",Istart,Iend); 
> 
> I get
> $ mpiexec -n 3 ./ex2
> Istart 0 Iend 19
> Istart 19 Iend 38
> Istart 38 Iend 56
> Norm of error 0.000522061 iterations 9
> The matrix global size = 56
> 
> Hong
> 
> Your replies are highly appreciated
> 
> Thanks,
> Maahi Talukder
> Clarkson University 
> 
> 


From bsmith at mcs.anl.gov  Fri Feb 15 17:32:03 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 15 Feb 2019 23:32:03 +0000
Subject: [petsc-users] KSP ex2.c and Parallelization
In-Reply-To: <CAJOaupGdWKh9w4aia3TJ9R=KVbXW8JAHkFh0fGhnbdCH7_hYQw@mail.gmail.com>
References: <CAJOaupF5jGqNMo-XdRiOS2BVSRVcv+B1RN2yLKe5AYEFJ5pBAw@mail.gmail.com>
	<CAGCphBvrgJd+reQRydVuGzjr6KUbdh8vUbCcJqAm67kovvPNtw@mail.gmail.com>
	<CAJOaupEeSBDQ05AAE-c=9AamL9AWr_BWRCP5r2y5nkZMHoD8LQ@mail.gmail.com>
	<8235A67A-7378-4A79-B5C9-8B48802D57AE@anl.gov>
	<CAJOaupGdWKh9w4aia3TJ9R=KVbXW8JAHkFh0fGhnbdCH7_hYQw@mail.gmail.com>
Message-ID: <C39698E9-66EA-4DB1-897B-CF7706929C36@mcs.anl.gov>


   Unfortunately converting a sequential code to MPI parallelism requires a great deal of restructuring of the code (often little of the original code can be retained) to get good performance. This is because the data has to be partitioned well in parallel to get good performance. src/ksp/ksp/examples/tutorials/ex2.c is not a good example because it doesn't make an effort to partition the data for good parallel performance.

   The PETSc DMDACreate2d() is designed to handle the partitioning of the data and generation of the matrix effectively in parallel. Take a look at, for example, src/ksp/ksp/examples/tutorials/ex46.c 

  Barry




> On Feb 15, 2019, at 5:03 PM, Maahi Talukder <maahi.buet at gmail.com> wrote:
> 
> 
> Thank you so much for your reply.
> 
>  I am trying to solve  the inverse of Poisson's equation to generate structured grid. The discretization is the 9-point finite difference.  The elements of my 'A' matrix are being calculated as it follows and being stored in Q matrix. I am planning to pass the values of Q() matrix to PETSc Mat M to solve my linear system. And I am trying to solve it in parallel, as my sequential code is generating results perfectly. So to implement in parallel, I was following the ex2.c . So what do you think about my approch? 
> 
> Do i = 2,ymax-1
> 
> Do j = 2,xmax-1
> 
> g11(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i,j+1)-x(i,j-1)) +0.25*(y(i,j+1)-y(i,j-1))*(y(i,j+1)-y(i,j-1))
> g12(i-1,j-1) = 0.25*(x(i,j+1)-x(i,j-1))*(x(i+1,j)-x(i-1,j))+ 0.25*(y(i,j+1)-y(i,j-1))*(y(i+1,j)-y(i-1,j))
> g22(i-1,j-1) = 0.25*(x(i+1,j)-x(i-1,j))*(x(i+1,j)-x(i-1,j)) + 0.25*(y(i+1,j)-y(i-1,j))*(y(i+1,j)-y(i-1,j))
> 
> M(j-1,j-1+xmax*(i-2))= (-1)*(g12(i-1,j-1)/2)
> M(j-1,j+xmax*(i-2))= g11(i-1,j-1)
> M(j-1,j+1+xmax*(i-2)) = g12(i-1,j-1)/2
> M(j-1,j-1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> M(j-1,j+xmax*(i-2)+xmax)= (-2)*(g22(i-1,j-1)+g11(i-1,j-1))
> M(j-1,j+1+xmax*(i-2)+xmax)= g22(i-1,j-1)
> M(j-1,j-1+xmax*(i-2)+2*xmax) = g12(i-1,j-1)/2
> M(j-1,j+xmax*(i-2)+2*xmax) = g11(i-1,j-1)
> M(j-1,j+1+xmax*(i-2)+2*xmax) = (-1)*(g12(i-1,j-1)/2)       
> 
> end Do
> 
> N(1+(xmax-2)*(i-2):(xmax-2)*(i-1),1:xmax*ymax)= M(1:xmax-2,1:xmax*ymax)
> M = 0
> 
> end Do
> 
> E(1:(xmax-2)*(ymax-2),1:xmax*ymax-2*xmax) = N(1:(xmax-2)*(ymax-2),xmax+1:xmax*ymax-xmax)
> 
> Do i = 1,ymax-2
>     
> Q(1:(xmax-2)*(ymax-2),1+(xmax-2)*(i-1):(xmax-2)*i)= E(1:(xmax-2)*(ymax-2),2+xmax*(i-1):(xmax-1)+xmax*(i-1))
>     
> end Do
> 
> Thanks,
> Maahi Talukder
> 
> On Fri, Feb 15, 2019 at 5:47 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>    Maybe we could provide more guidance on how to proceed if we knew a bit more about the 1) problem you are trying to solve, 2) the discretization you are using and 3) the mesh you are using?
> 
>     From your code fragment with a m*n I am guessing you are solving something on a structured grid with m grid points in the x direction and n grid points in the y direction? 
> 
>   Barry
> 
> 
> > On Feb 15, 2019, at 4:26 PM, Maahi Talukder via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Thank you so much for your reply. 
> >  
> > But in the definition of d_nz of MatMPIAIJSetPreallocation  , it is defined as 'number of nonzeros per row in DIAGONAL portion of local submatrix'. So I was wondering about how to find the Diagonal portion of the local submatrix that PETSc is splitting the big matrix into . So should it be the maximum number of nonzeros in each row of the big matrix instead of  the min number of nonzeros in the diagonal portion of the local matrix?
> > 
> > And so MatGetOwnershipRange returns the start and the end of all the smaller matrices that PETSc is splitting the big matrix into, right?
> > 
> > 
> > Regards,
> > Maahi Talukder
> > 
> > On Fri, Feb 15, 2019 at 2:43 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
> > Maahi :
> > Hello All,
> > 
> > I have some questions regarding ex2.c in /petsc/src/ksp/ksp/examples/tutorials/ex2.c. 
> > 
> > 1.   The  local sizes of the matrix was decided by PETSc using ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
> > But without knowing the local sizes, how was the following function used which requires the knowledge of how the big matrix was split into smaller chunks within PETSc? 
> >   ierr = MatMPIAIJSetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);
> > This example sets a matrix for 2D 5-point PDE stencil, for which the maximum non-zeros in each row is 5. 
> > 
> > 2.  When using  MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr),  the global index of  first and last row of which local chunk of the big matrix is it returning ? I mean if the big matrix was split into three processes, for example, fist and last global row of which process is it returning? 
> > Adding a line 
> >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > +  printf("Istart %d Iend %d\n",Istart,Iend); 
> > 
> > I get
> > $ mpiexec -n 3 ./ex2
> > Istart 0 Iend 19
> > Istart 19 Iend 38
> > Istart 38 Iend 56
> > Norm of error 0.000522061 iterations 9
> > The matrix global size = 56
> > 
> > Hong
> > 
> > Your replies are highly appreciated
> > 
> > Thanks,
> > Maahi Talukder
> > Clarkson University 
> > 
> > 
> 


From mbuerkle at web.de  Fri Feb 15 18:49:18 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Sat, 16 Feb 2019 01:49:18 +0100
Subject: [petsc-users] superlu_dist+parmetis
In-Reply-To: <CAFvbobWGfhkuuhPSXJT0G+uBU_RxbuJYgxwPn1Y-49Q3bO9Ozg@mail.gmail.com>
References: <trinity-61915993-adfa-45c9-97f9-33c43d98de67-1550191499056@3c-app-webde-bs42>
	<55EA92E9-9A7F-45C0-A009-7062EF89E4D4@anl.gov>
	<trinity-72ef7874-70fe-4eb1-a167-757b11df22b0-1550212211771@3c-app-webde-bs19>
	<5A25FA95-0301-470D-8B4A-813ADFCB9C72@mcs.anl.gov>
	<CAFvbobWGfhkuuhPSXJT0G+uBU_RxbuJYgxwPn1Y-49Q3bO9Ozg@mail.gmail.com>
Message-ID: <trinity-eab522dc-47d7-4a07-912a-7fcf185c1a54-1550278158080@3c-app-webde-bap62>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190216/82ff85e8/attachment.html>

From tempohoper at gmail.com  Mon Feb 18 09:47:34 2019
From: tempohoper at gmail.com (Sal Am)
Date: Mon, 18 Feb 2019 15:47:34 +0000
Subject: [petsc-users] saving results
Message-ID: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>

Is there a function/command line option to save the solution as it is
solving (and read in the file from where it crashed and keep iterating from
there perhaps)?
Had a seg fault and all the results until that point seems to have been
lost.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/72e3d599/attachment.html>

From jed at jedbrown.org  Mon Feb 18 10:10:14 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 18 Feb 2019 09:10:14 -0700
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
Message-ID: <8736olhyc9.fsf@jedbrown.org>

What kind of solver are you using and how often do you want to write?

Sal Am via petsc-users <petsc-users at mcs.anl.gov> writes:

> Is there a function/command line option to save the solution as it is
> solving (and read in the file from where it crashed and keep iterating from
> there perhaps)?
> Had a seg fault and all the results until that point seems to have been
> lost.

From tempohoper at gmail.com  Mon Feb 18 10:19:29 2019
From: tempohoper at gmail.com (Sal Am)
Date: Mon, 18 Feb 2019 16:19:29 +0000
Subject: [petsc-users] saving results
In-Reply-To: <8736olhyc9.fsf@jedbrown.org>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<8736olhyc9.fsf@jedbrown.org>
Message-ID: <CAE3k7C7ZZdY=ULCUfHXFxYLiqpeTZafc5_=Nu-y_QDZ8gmBNmg@mail.gmail.com>

This is how I run it:

-ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
-build_twosided allreduce -ksp_monitor_true_residual
-ksp_monitor_if_not_converged -log_view -ksp_max_it 100000 -ksp_rtol 1.0e-7

so BCGS with GAMG as preconditioner. I am guessing writing at every
timestep would be expensive, maybe every hour? I am not sure what would be
a good number here if the simulations lasts more than a day.



On Mon, Feb 18, 2019 at 4:10 PM Jed Brown <jed at jedbrown.org> wrote:

> What kind of solver are you using and how often do you want to write?
>
> Sal Am via petsc-users <petsc-users at mcs.anl.gov> writes:
>
> > Is there a function/command line option to save the solution as it is
> > solving (and read in the file from where it crashed and keep iterating
> from
> > there perhaps)?
> > Had a seg fault and all the results until that point seems to have been
> > lost.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/6ba9b6c5/attachment.html>

From jed at jedbrown.org  Mon Feb 18 10:26:17 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 18 Feb 2019 09:26:17 -0700
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C7ZZdY=ULCUfHXFxYLiqpeTZafc5_=Nu-y_QDZ8gmBNmg@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<8736olhyc9.fsf@jedbrown.org>
	<CAE3k7C7ZZdY=ULCUfHXFxYLiqpeTZafc5_=Nu-y_QDZ8gmBNmg@mail.gmail.com>
Message-ID: <87wolxgj12.fsf@jedbrown.org>

What kind of problems are you solving?  Running for a Krylov method for
tens of thousands of iterations is very rarely recommended.

Regarding storage, it's significantly more expensive to store the Krylov
basis (even when it's a recurrence) than the current approximation.
Some methods require some work to compute the current approximation.
Anyway, you can use a KSPMonitor to write checkpoints at whatever
interval you like.  In case of a crash, use VecLoad() to read in the
checkpoint file and use KSPSetInitialGuessNonzero() to make it be used
as an initial guess.

Sal Am <tempohoper at gmail.com> writes:

> This is how I run it:
>
> -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
> -build_twosided allreduce -ksp_monitor_true_residual
> -ksp_monitor_if_not_converged -log_view -ksp_max_it 100000 -ksp_rtol 1.0e-7
>
> so BCGS with GAMG as preconditioner. I am guessing writing at every
> timestep would be expensive, maybe every hour? I am not sure what would be
> a good number here if the simulations lasts more than a day.
>
>
>
> On Mon, Feb 18, 2019 at 4:10 PM Jed Brown <jed at jedbrown.org> wrote:
>
>> What kind of solver are you using and how often do you want to write?
>>
>> Sal Am via petsc-users <petsc-users at mcs.anl.gov> writes:
>>
>> > Is there a function/command line option to save the solution as it is
>> > solving (and read in the file from where it crashed and keep iterating
>> from
>> > there perhaps)?
>> > Had a seg fault and all the results until that point seems to have been
>> > lost.
>>

From bsmith at mcs.anl.gov  Mon Feb 18 14:21:18 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 18 Feb 2019 20:21:18 +0000
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
Message-ID: <FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>


   I'm guessing you mange your own time stepping (and nonlinear solver if there is one)? 

   You can save the solution with a call to VecView() and then reload the solution with a VecLoad() but you need to manage any other restart data that you may need, like the value of the current time etc. 

   As Jed says it doesn't make sense to save "partial" solutions within the linear solver.  Just save it every several time-steps.

   Barry

   Also the code should not be "crashing" at seemingly long times (after hours) with a Segmentation fault. Send us the full error message and we'll see if there is some way we can help you fix this problem.


> On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Is there a function/command line option to save the solution as it is solving (and read in the file from where it crashed and keep iterating from there perhaps)?
> Had a seg fault and all the results until that point seems to have been lost. 
> 
> 
> 
> 


From rlmackie862 at gmail.com  Mon Feb 18 19:42:00 2019
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Mon, 18 Feb 2019 17:42:00 -0800
Subject: [petsc-users] error with ASM and GPUs
Message-ID: <0B4F0636-F3DC-48D7-A9DF-AEF8FDF190D7@gmail.com>

We are investigating the use of porting our PETSc code to work on GPUS.
We have made some good progress but are encountering and error when specifying ASM as the preconditioner.

Specifically, we set:

-ksp_type bcgs
-pc_type asm
-sub_pc_type ilu

That bombs out with the message below.


  Residual norms for em_ solve.
  0 KSP preconditioned resid norm 9.464272188534e+06 true resid norm 2.587899098807e+02 ||r(i)||/||b|| 1.000000000000e+00
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
[0]PETSC ERROR: /home/everderio/DEV/bin/mtd3fwd_real_debug on a linux-intel-intelmpi-real-debug-cuda_2018 named GPU by root Mon Feb 18 16:46:14 2019
[0]PETSC ERROR: Configure options --with-clean=1 --with-debugging=1 --with-fortran=1 --with-blaslapack-dir=/opt/intel/mkl --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc
[0]PETSC ERROR: #1 VecCUDAGetArrayReadWrite() line 1210 in /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: #2 VecScatterCUDA_StoS() line 272 in /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/vecscattercuda.cu
[0]PETSC ERROR: #3 VecScatterBegin_SGToSS_Stride1() line 384 in /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/impls/vscat.c
[0]PETSC ERROR: #4 VecScatterBegin() line 110 in /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/interface/vscatfce.c
[0]PETSC ERROR: #5 PCApply_ASM() line 486 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/impls/asm/asm.c
[0]PETSC ERROR: #6 PCApply() line 462 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #7 PCApplyBAorAB() line 691 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #8 KSP_PCApplyBAorAB() line 309 in /home/everderio/DEV/petsc-3.10.3/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #9 KSPSolve_BCGS() line 87 in /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: #10 KSPSolve() line 780 in /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
 

However, using the following options work:

-ksp_type bcgs
-pc_type bjacobi
-sub_pc_type ilu


Thanks, Randy M.


From yyang85 at stanford.edu  Mon Feb 18 20:08:35 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Tue, 19 Feb 2019 02:08:35 +0000
Subject: [petsc-users] PetscBinaryRead fails
Message-ID: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello,

I am writing a very simple function just to read in a single value from an input file (which also just contains one single scalar value). Using PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error. Could you help me figure out why?

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Read from file failed
[0]PETSC ERROR: Read past end of file
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
[0]PETSC ERROR: ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis --download-hypre --download-superlu --download-mumps --download-scalapack --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
[0]PETSC ERROR: #1 PetscBinaryRead() line 276 in /home/yyy910805/petsc/src/sys/fileio/sysio.c
0.000000

Thanks a lot!
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/56299f61/attachment.html>

From knepley at gmail.com  Mon Feb 18 20:18:18 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 18 Feb 2019 21:18:18 -0500
Subject: [petsc-users] error with ASM and GPUs
In-Reply-To: <0B4F0636-F3DC-48D7-A9DF-AEF8FDF190D7@gmail.com>
References: <0B4F0636-F3DC-48D7-A9DF-AEF8FDF190D7@gmail.com>
Message-ID: <CAMYG4Gkq32z9vTN3-x_TkumL8SmDAx3ZbJjUpaXSNBJuL0zWkw@mail.gmail.com>

On Mon, Feb 18, 2019 at 8:43 PM Randall Mackie via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> We are investigating the use of porting our PETSc code to work on GPUS.
> We have made some good progress but are encountering and error when
> specifying ASM as the preconditioner.
>
> Specifically, we set:
>
> -ksp_type bcgs
> -pc_type asm
> -sub_pc_type ilu
>
> That bombs out with the message below.
>

It certainly looks like BJACOBI propagates the matrix type correctly to
submatrices, and ASM does not.
Patrick fixed this for BJacobi I believe. Patrick, do you remember what
ChangeSet fixed it?

  Thanks,

     Matt


>
>   Residual norms for em_ solve.
>   0 KSP preconditioned resid norm 9.464272188534e+06 true resid norm
> 2.587899098807e+02 ||r(i)||/||b|| 1.000000000000e+00
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: /home/everderio/DEV/bin/mtd3fwd_real_debug on a
> linux-intel-intelmpi-real-debug-cuda_2018 named GPU by root Mon Feb 18
> 16:46:14 2019
> [0]PETSC ERROR: Configure options --with-clean=1 --with-debugging=1
> --with-fortran=1 --with-blaslapack-dir=/opt/intel/mkl --with-cuda=1
> --with-cudac=/usr/local/cuda-10.0/bin/nvcc
> [0]PETSC ERROR: #1 VecCUDAGetArrayReadWrite() line 1210 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: #2 VecScatterCUDA_StoS() line 272 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/
> vecscattercuda.cu
> [0]PETSC ERROR: #3 VecScatterBegin_SGToSS_Stride1() line 384 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/impls/vscat.c
> [0]PETSC ERROR: #4 VecScatterBegin() line 110 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/interface/vscatfce.c
> [0]PETSC ERROR: #5 PCApply_ASM() line 486 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/impls/asm/asm.c
> [0]PETSC ERROR: #6 PCApply() line 462 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #7 PCApplyBAorAB() line 691 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #8 KSP_PCApplyBAorAB() line 309 in
> /home/everderio/DEV/petsc-3.10.3/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #9 KSPSolve_BCGS() line 87 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
> [0]PETSC ERROR: #10 KSPSolve() line 780 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>
>
> However, using the following options work:
>
> -ksp_type bcgs
> -pc_type bjacobi
> -sub_pc_type ilu
>
>
> Thanks, Randy M.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/101495b9/attachment.html>

From knepley at gmail.com  Mon Feb 18 20:25:06 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 18 Feb 2019 21:25:06 -0500
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4GmvYOCtu4KcqaSBAMqd9cqv1GXs0UncyvvNm8vSxCwe6w@mail.gmail.com>

It sounds like you opened the file for writing.

   Matt

On Mon, Feb 18, 2019 at 9:08 PM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
>
>
> I am writing a very simple function just to read in a single value from an
> input file (which also just contains one single scalar value). Using
> PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error.
> Could you help me figure out why?
>
>
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Read from file failed
>
> [0]PETSC ERROR: Read past end of file
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
>
> [0]PETSC ERROR: ./output on a arch-linux2-c-debug named
> DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-shared-libraries=1 --download-hdf5 --download-metis
> --download-parmetis --download-hypre --download-superlu --download-mumps
> --download-scalapack --download-mpich
> --download-mpich-configure-arguments="--enable-error-messages=all
> --enable-g"
>
> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in
> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>
> 0.000000
>
>
>
> Thanks a lot!
>
> Yuyun
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/523f401c/attachment-0001.html>

From knepley at gmail.com  Mon Feb 18 20:27:09 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 18 Feb 2019 21:27:09 -0500
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <CAMYG4GmvYOCtu4KcqaSBAMqd9cqv1GXs0UncyvvNm8vSxCwe6w@mail.gmail.com>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4GmvYOCtu4KcqaSBAMqd9cqv1GXs0UncyvvNm8vSxCwe6w@mail.gmail.com>
Message-ID: <CAMYG4Gnzx473_im6_r4RgM57CS180qbBLG9JO_kz8noJK+NsHw@mail.gmail.com>

If you used a PetscVIewer(), then you need to set the mode


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerFileSetMode.html

or open it for reading with PETSC_MODE_READ


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html

  Thanks,

    Matt

On Mon, Feb 18, 2019 at 9:25 PM Matthew Knepley <knepley at gmail.com> wrote:

> It sounds like you opened the file for writing.
>
>    Matt
>
> On Mon, Feb 18, 2019 at 9:08 PM Yuyun Yang via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello,
>>
>>
>>
>> I am writing a very simple function just to read in a single value from
>> an input file (which also just contains one single scalar value). Using
>> PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error.
>> Could you help me figure out why?
>>
>>
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>>
>> [0]PETSC ERROR: Read from file failed
>>
>> [0]PETSC ERROR: Read past end of file
>>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>>
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
>>
>> [0]PETSC ERROR: ./output on a arch-linux2-c-debug named
>> DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
>>
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-shared-libraries=1 --download-hdf5 --download-metis
>> --download-parmetis --download-hypre --download-superlu --download-mumps
>> --download-scalapack --download-mpich
>> --download-mpich-configure-arguments="--enable-error-messages=all
>> --enable-g"
>>
>> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in
>> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>>
>> 0.000000
>>
>>
>>
>> Thanks a lot!
>>
>> Yuyun
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/f91897ef/attachment.html>

From bsmith at mcs.anl.gov  Mon Feb 18 20:41:17 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 19 Feb 2019 02:41:17 +0000
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <69CB0A51-06DB-4D8E-8647-BCD16AC031BD@anl.gov>


  Send program and data file if you cannot get it to work.


> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
>  
> I am writing a very simple function just to read in a single value from an input file (which also just contains one single scalar value). Using PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error. Could you help me figure out why?
>  
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Read from file failed
> [0]PETSC ERROR: Read past end of file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
> [0]PETSC ERROR: ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis --download-hypre --download-superlu --download-mumps --download-scalapack --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in /home/yyy910805/petsc/src/sys/fileio/sysio.c
> 0.000000
>  
> Thanks a lot!
> Yuyun


From yyang85 at stanford.edu  Mon Feb 18 21:23:49 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Tue, 19 Feb 2019 03:23:49 +0000
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <69CB0A51-06DB-4D8E-8647-BCD16AC031BD@anl.gov>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<69CB0A51-06DB-4D8E-8647-BCD16AC031BD@anl.gov>
Message-ID: <DM6PR02MB5242DCCCA64277CF9FDE2EE5D27C0@DM6PR02MB5242.namprd02.prod.outlook.com>

I used FILE_MODE_READ. Please see attached file (the function is part of another file therefore all the other include statements; I've extracted it here). The test file only contains one number. My directory is hard-coded into the main function.

Thanks a lot!
Yuyun

-----Original Message-----
From: Smith, Barry F. <bsmith at mcs.anl.gov> 
Sent: Monday, February 18, 2019 6:41 PM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] PetscBinaryRead fails


  Send program and data file if you cannot get it to work.


> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello,
>  
> I am writing a very simple function just to read in a single value from an input file (which also just contains one single scalar value). Using PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error. Could you help me figure out why?
>  
> [0]PETSC ERROR: --------------------- Error Message 
> --------------------------------------------------------------
> [0]PETSC ERROR: Read from file failed
> [0]PETSC ERROR: Read past end of file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: 
> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu 
> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis --download-hypre --download-superlu --download-mumps --download-scalapack --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
> /home/yyy910805/petsc/src/sys/fileio/sysio.c
> 0.000000
>  
> Thanks a lot!
> Yuyun

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testCheckpointLoadVal.cpp
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/12bf74f2/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: _test
Type: application/octet-stream
Size: 6 bytes
Desc: _test
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/12bf74f2/attachment.obj>

From bsmith at mcs.anl.gov  Mon Feb 18 22:06:16 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 19 Feb 2019 04:06:16 +0000
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <DM6PR02MB5242DCCCA64277CF9FDE2EE5D27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<69CB0A51-06DB-4D8E-8647-BCD16AC031BD@anl.gov>
	<DM6PR02MB5242DCCCA64277CF9FDE2EE5D27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <6F3BC8C7-6D00-441C-9472-B315214EAD74@mcs.anl.gov>


  I don't know how you got your _test file but it is the wrong size

$ ls -l _test
-rw-r--r--@ 1 barrysmith  staff  6 Feb 18 21:25 _test
~/Src/petsc/test-dir (maint=) arch-basic

it only has 6 bytes but to hold a PETSC_SCALAR (when PetscScalar is real) it should have 8 bytes. For example the code

  PetscViewer viewer;
  PetscScalar v;
  int fd;
  
      PetscViewerBinaryOpen(PETSC_COMM_WORLD, "joe", FILE_MODE_WRITE, &viewer);
    PetscViewerBinaryGetDescriptor(viewer, &fd);
    PetscBinaryWrite(fd, &value, 1, PETSC_SCALAR,PETSC_FALSE);
    PetscViewerDestroy(&viewer);

produces a file with 8 bytes in it

$ ls -l joe
-rw-r--r--  1 barrysmith  staff  8 Feb 18 22:03 joe

which can be read in by your program.

> On Feb 18, 2019, at 9:23 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
> 
> I used FILE_MODE_READ. Please see attached file (the function is part of another file therefore all the other include statements; I've extracted it here). The test file only contains one number. My directory is hard-coded into the main function.
> 
> Thanks a lot!
> Yuyun
> 
> -----Original Message-----
> From: Smith, Barry F. <bsmith at mcs.anl.gov> 
> Sent: Monday, February 18, 2019 6:41 PM
> To: Yuyun Yang <yyang85 at stanford.edu>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PetscBinaryRead fails
> 
> 
>  Send program and data file if you cannot get it to work.
> 
> 
>> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> Hello,
>> 
>> I am writing a very simple function just to read in a single value from an input file (which also just contains one single scalar value). Using PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error. Could you help me figure out why?
>> 
>> [0]PETSC ERROR: --------------------- Error Message 
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Read from file failed
>> [0]PETSC ERROR: Read past end of file
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: 
>> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu 
>> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis --download-hypre --download-superlu --download-mumps --download-scalapack --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
>> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
>> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>> 0.000000
>> 
>> Thanks a lot!
>> Yuyun
> 
> <testCheckpointLoadVal.cpp><_test>


From yyang85 at stanford.edu  Mon Feb 18 23:14:26 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Tue, 19 Feb 2019 05:14:26 +0000
Subject: [petsc-users] PetscBinaryRead fails
In-Reply-To: <6F3BC8C7-6D00-441C-9472-B315214EAD74@mcs.anl.gov>
References: <DM6PR02MB524249EE68D99E65F98C4B9DD27C0@DM6PR02MB5242.namprd02.prod.outlook.com>
	<69CB0A51-06DB-4D8E-8647-BCD16AC031BD@anl.gov>
	<DM6PR02MB5242DCCCA64277CF9FDE2EE5D27C0@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<6F3BC8C7-6D00-441C-9472-B315214EAD74@mcs.anl.gov>
Message-ID: <DM6PR02MB5242C15295E538EDD8A2F414D27C0@DM6PR02MB5242.namprd02.prod.outlook.com>

Oh I see, thanks for confirming it works with an 8-byte file. I didn't generate it from a code but just put a random number in a blank file. I thought it would automatically be the right size but that isn't the case.


Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Monday, February 18, 2019 8:06:16 PM
To: Yuyun Yang
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] PetscBinaryRead fails


  I don't know how you got your _test file but it is the wrong size

$ ls -l _test
-rw-r--r--@ 1 barrysmith  staff  6 Feb 18 21:25 _test
~/Src/petsc/test-dir (maint=) arch-basic

it only has 6 bytes but to hold a PETSC_SCALAR (when PetscScalar is real) it should have 8 bytes. For example the code

  PetscViewer viewer;
  PetscScalar v;
  int fd;

      PetscViewerBinaryOpen(PETSC_COMM_WORLD, "joe", FILE_MODE_WRITE, &viewer);
    PetscViewerBinaryGetDescriptor(viewer, &fd);
    PetscBinaryWrite(fd, &value, 1, PETSC_SCALAR,PETSC_FALSE);
    PetscViewerDestroy(&viewer);

produces a file with 8 bytes in it

$ ls -l joe
-rw-r--r--  1 barrysmith  staff  8 Feb 18 22:03 joe

which can be read in by your program.

> On Feb 18, 2019, at 9:23 PM, Yuyun Yang <yyang85 at stanford.edu> wrote:
>
> I used FILE_MODE_READ. Please see attached file (the function is part of another file therefore all the other include statements; I've extracted it here). The test file only contains one number. My directory is hard-coded into the main function.
>
> Thanks a lot!
> Yuyun
>
> -----Original Message-----
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Monday, February 18, 2019 6:41 PM
> To: Yuyun Yang <yyang85 at stanford.edu>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PetscBinaryRead fails
>
>
>  Send program and data file if you cannot get it to work.
>
>
>> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>
>> Hello,
>>
>> I am writing a very simple function just to read in a single value from an input file (which also just contains one single scalar value). Using PetscBinaryRead(fd, &value, 1, PETSC_SCALAR) gives me the following error. Could you help me figure out why?
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Read from file failed
>> [0]PETSC ERROR: Read past end of file
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR:
>> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu
>> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis --download-hypre --download-superlu --download-mumps --download-scalapack --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
>> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in
>> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>> 0.000000
>>
>> Thanks a lot!
>> Yuyun
>
> <testCheckpointLoadVal.cpp><_test>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/cf92b104/attachment.html>

From j.wagner at rice.edu  Tue Feb 19 00:11:08 2019
From: j.wagner at rice.edu (Jordan Wagner)
Date: Tue, 19 Feb 2019 00:11:08 -0600
Subject: [petsc-users] Including constrained dofs in a matrix/vector created
 with DMCreateXXXX()
Message-ID: <be667eba-cec1-27be-02bf-878f3090fb07@rice.edu>

Hi,

Over the past few months, I have implemented dmplex/section in my 
preexisting finite element code. We already had our own implementations 
for things like the finite element space, boundary conditions, 
projections, etc, so I am mostly using dmplex/section at its lowest 
level; no DT/DS stuff for the moment. Everything is working fine at this 
level, and I am overall very happy with how it ties into the code that 
we already have.

I am currently doing my own preallocation for the system Jacobian but am 
wanting to use DMCreateMatrix() instead, as it seems pointless to keep 
using my own crappy function when this one is just sitting there 
waiting. However, I dislike how I am currently implementing this and was 
hoping to get some suggestions or some advice. My problem is handling 
nicely the Dirichlet dofs. My code takes the approach of first 
assembling the entire Jacobian matrix and load vector as if no 
constraints are imposed. We then have a function that applies the 
essential conditions after assembly and extracts a submatrix and 
subvector similar to the way ex49 
<https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex49.c.html> 
is doing.

Since DMCreateMatrix() automatically partitions out the constraints 
specified in the section, I find myself having to create two nearly 
identical sections: one that has constraints and one that does not, and 
then I clone the DM and set the default section of each respectively. I 
can then use DMCreateMatrix() on the one dm/section with no constraints 
to preallocate the larger matrix that I am assembling. Then I use the 
dm/section with constraints to do all of my boundary condition and 
projection operations on the previously created matrix and vector. 
Essentially, I am using an unconstrained section only for allocation and 
a constrained section for most everything else.

This process actually works fine for me, but its pretty ugly, and I am 
sure there is a better way. I am wondering if I am missing something 
that could keep me from having to go through this process of creating 
two sections that differ only in the constraints. It seems to me if 
there were an option for DMCreateMatrix()/DMCreateVector() to either 
include or not include the dofs, that would completely solve my problem. 
That way, I can use the same section for both creation and allocation.

So, the main question I have is just whether there is a function or 
option that I am missing like DMCreateXXXX() but with the ability to 
retain the constrained rows and columns rather than compressing them out.

Thanks a lot, as always.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/482763a4/attachment.html>

From tempohoper at gmail.com  Tue Feb 19 03:11:25 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 19 Feb 2019 09:11:25 +0000
Subject: [petsc-users] saving results
In-Reply-To: <FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
Message-ID: <CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>

 It is a finite-element problem of an RF antenna dielectric interaction
where all the non-zero elements are on the diagonal of the sparse matrix
(if that is relevant).
More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor, it
seems that I have to write my own routine?

> Running for a Krylov method for tens of thousands of iterations is very
> rarely recommended.
>
GAMG and BCGS is the only ones that have actually worked for me so far, I
increased the max_it because it was not enough with the default one. With
the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
need more accuracy so I increased the tol to 10^-7 and that is when it
crashed after ~51000 iterations.

@Barry

> I'm guessing you mange your own time stepping (and nonlinear solver if
> there is one)?
>

It is the default from the solver (attached the short code).

> As Jed says it doesn't make sense to save "partial" solutions within the
> linear solver.  Just save it every several time-steps.
>

Yes maybe I worded it wrong. I did mean to save checkpoints basically such
that the next time it is running it can pick up from where it crashed.

   Also the code should not be "crashing" at seemingly long times (after
> hours) with a Segmentation fault. Send us the full error message and we'll
> see if there is some way we can help you fix this problem.
>
I have added this before, but seemingly they conclusion was that it is
memory error...although I am not sure how, but I have added the entire
error.

On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    I'm guessing you mange your own time stepping (and nonlinear solver if
> there is one)?
>
>    You can save the solution with a call to VecView() and then reload the
> solution with a VecLoad() but you need to manage any other restart data
> that you may need, like the value of the current time etc.
>
>    As Jed says it doesn't make sense to save "partial" solutions within
> the linear solver.  Just save it every several time-steps.
>
>    Barry
>
>    Also the code should not be "crashing" at seemingly long times (after
> hours) with a Segmentation fault. Send us the full error message and we'll
> see if there is some way we can help you fix this problem.
>
>
> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Is there a function/command line option to save the solution as it is
> solving (and read in the file from where it crashed and keep iterating from
> there perhaps)?
> > Had a seg fault and all the results until that point seems to have been
> lost.
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/980421ce/attachment-0001.html>
-------------- next part --------------
static char help[] = "Solves a complex sparse linear system in parallel with KSP.\n\n";

#include <petscksp.h>
#include <iostream>
int main(int argc,char **args){
    Vec            x,b;           /* approx solution, RHS */
    Mat            A, F;             /* linear system matrix */
    KSP            ksp;           /* linear solver context */
    // PetscReal      norm;          /* norm of solution error */
    PC             pc;            /* petsc preconditioner object */
    PetscMPIInt    rank, size;    /* petsc mpi rank and size object */
    PetscViewer    viewer;        /* petsc viewer object needed for vis. and reading */
    PetscErrorCode ierr;

    // Initialise PETSc needed in the beginning of program, includes MPI initialisation. 
    ierr = PetscInitialize(&argc,&args,(char*)0,help);if (ierr) return ierr;
    MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
    MPI_Comm_size(PETSC_COMM_WORLD,&size);

    // Check if configured with complex
    #if !defined(PETSC_USE_COMPLEX)
      SETERRQ(PETSC_COMM_WORLD,1,"This example requires complex numbers");
    #endif
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
           Read the matrix and right-hand-side vector that defines
           the linear system, Ax = b.
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Reading vector in binary from Vector_b.dat ...\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"../petscpy/petscFiles/Vector_b.dat",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
    ierr = VecCreate(PETSC_COMM_WORLD, &b);CHKERRQ(ierr);
    ierr = VecLoad(b,viewer); CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Vector b read.\n");CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);

    ierr = PetscPrintf(PETSC_COMM_WORLD,"Reading matrix in binary from Matrix_A.dat ...\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"../petscpy/petscFiles/Matrix_A.dat",FILE_MODE_READ,&viewer);CHKERRQ(ierr);
    ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
    ierr = MatLoad(A,viewer);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Matrix A read.\n");CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
    // Duplicate vector b for solution x, content not duplicated.
    ierr = VecDuplicate(b,&x);CHKERRQ(ierr);
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                  Create the linear solver and set various options
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

    
    // Create linear solver context
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Creating KSPCreate\n");CHKERRQ(ierr);
    ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);

    /* Set operators. Here the matrix that defines the linear system
       also serves as the preconditioning constructor matrix.*/
    ierr = PetscPrintf(PETSC_COMM_WORLD,"KSPSetOperators initiated\n");CHKERRQ(ierr);
    ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr);

    /* Set runtime options, e.g., -ksp_type <type> -pc_type <type>
     -ksp_monitor -ksp_rtol <rtol>. Here MUMPS is used if no runtime options is provided. */
    // ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr);
    // ierr = KSPGetPC(ksp, &pc);CHKERRQ(ierr);
    // ierr = PCSetType(pc, PCLU);CHKERRQ(ierr);
    // ierr = PCFactorSetMatSolverType(pc, MATSOLVERMUMPS);CHKERRQ(ierr);
    ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);

    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                        Solve the linear system
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    /* KSPSetUp and KSPSetUpOnBlocks are optional. 
    Used to enable more precise profiling of setting up the preconditioner */ 
    ierr = PetscPrintf(PETSC_COMM_WORLD,"KSPSetUp initiated\n");CHKERRQ(ierr);
    ierr = KSPSetUp(ksp);CHKERRQ(ierr);
    // ierr = PCFactorGetMatrix(pc, &F);
    //ierr = KSPSetUpOnBlocks(ksp);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Solving the system Ax=b\n");CHKERRQ(ierr);
    ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Done solving.\n");CHKERRQ(ierr);

    // Uncomment the following to view the solution in terminal
    // ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);
    /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                        Save solution and clean up
       - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
    ierr = PetscPrintf(PETSC_COMM_WORLD, "Writing solution x in PETSc binary file Vector_x_petsc.dat\n");CHKERRQ(ierr);
    ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "../petscpy/petscFiles/Vector_x_petsc.dat", FILE_MODE_WRITE, &viewer);CHKERRQ(ierr);
    ierr = PetscViewerBinarySetSkipHeader(viewer, PETSC_TRUE);CHKERRQ(ierr);
    ierr = VecView(x,viewer);CHKERRQ(ierr);CHKERRQ(ierr);
    ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr);
    /*
       Free work space.  All PETSc objects should be destroyed when they
       are no longer needed.
    */
    ierr = KSPDestroy(&ksp);CHKERRQ(ierr);
    ierr = VecDestroy(&x);CHKERRQ(ierr);
    ierr = VecDestroy(&b);CHKERRQ(ierr); 
    ierr = MatDestroy(&A);CHKERRQ(ierr);
    ierr = PetscFinalize();
    return ierr;
}


/*TEST

   build:
      requires: complex, MUMPS
      optional: SuperLU_dist, SuperLU

   test:
      mpiexec -n 4 ./test -ksp_type gmres -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual

TEST*/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OneStripAntenna.e42955
Type: application/octet-stream
Size: 472700 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/980421ce/attachment-0001.obj>

From mbuerkle at web.de  Tue Feb 19 05:15:21 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Tue, 19 Feb 2019 12:15:21 +0100
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
Message-ID: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/81d180fa/attachment.html>

From knepley at gmail.com  Tue Feb 19 06:15:40 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 07:15:40 -0500
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
	<CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
Message-ID: <CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>

On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> It is a finite-element problem of an RF antenna dielectric interaction
> where all the non-zero elements are on the diagonal of the sparse matrix
> (if that is relevant).
>

I am unsure whether this is what you mean. If you matrix is diagonal, you
can invert it exactly in a single step so you
do not need a Krylov solver. What equations are you solving? Maxwell?
Electrostatic? What finite element are you using?
  Thanks,

    Matt


> More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor, it
> seems that I have to write my own routine?
>
>> Running for a Krylov method for tens of thousands of iterations is very
>> rarely recommended.
>>
> GAMG and BCGS is the only ones that have actually worked for me so far, I
> increased the max_it because it was not enough with the default one. With
> the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
> need more accuracy so I increased the tol to 10^-7 and that is when it
> crashed after ~51000 iterations.
>
> @Barry
>
>> I'm guessing you mange your own time stepping (and nonlinear solver if
>> there is one)?
>>
>
> It is the default from the solver (attached the short code).
>
>> As Jed says it doesn't make sense to save "partial" solutions within the
>> linear solver.  Just save it every several time-steps.
>>
>
> Yes maybe I worded it wrong. I did mean to save checkpoints basically such
> that the next time it is running it can pick up from where it crashed.
>
>    Also the code should not be "crashing" at seemingly long times (after
>> hours) with a Segmentation fault. Send us the full error message and we'll
>> see if there is some way we can help you fix this problem.
>>
> I have added this before, but seemingly they conclusion was that it is
> memory error...although I am not sure how, but I have added the entire
> error.
>
> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>>    I'm guessing you mange your own time stepping (and nonlinear solver if
>> there is one)?
>>
>>    You can save the solution with a call to VecView() and then reload the
>> solution with a VecLoad() but you need to manage any other restart data
>> that you may need, like the value of the current time etc.
>>
>>    As Jed says it doesn't make sense to save "partial" solutions within
>> the linear solver.  Just save it every several time-steps.
>>
>>    Barry
>>
>>    Also the code should not be "crashing" at seemingly long times (after
>> hours) with a Segmentation fault. Send us the full error message and we'll
>> see if there is some way we can help you fix this problem.
>>
>>
>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > Is there a function/command line option to save the solution as it is
>> solving (and read in the file from where it crashed and keep iterating from
>> there perhaps)?
>> > Had a seg fault and all the results until that point seems to have been
>> lost.
>> >
>> >
>> >
>> >
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/ff7bdf1a/attachment.html>

From tempohoper at gmail.com  Tue Feb 19 07:21:43 2019
From: tempohoper at gmail.com (Sal Am)
Date: Tue, 19 Feb 2019 13:21:43 +0000
Subject: [petsc-users] saving results
In-Reply-To: <CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
	<CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
	<CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>
Message-ID: <CAE3k7C4fTAOfmp7Z5jYVOqR-hdesdks3=BdQPtp+DSU7S852yA@mail.gmail.com>

Matthew maybe indeed I was not clear and wrong in the wording. What I meant
is the matrix is undefined, but since it is a finite element method (using
second order elements). The pattern is like this:
[image: image.png]
We are indeed solving Maxwell's equations. I have added the paper.

On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> It is a finite-element problem of an RF antenna dielectric interaction
>> where all the non-zero elements are on the diagonal of the sparse matrix
>> (if that is relevant).
>>
>
> I am unsure whether this is what you mean. If you matrix is diagonal, you
> can invert it exactly in a single step so you
> do not need a Krylov solver. What equations are you solving? Maxwell?
> Electrostatic? What finite element are you using?
>   Thanks,
>
>     Matt
>
>
>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor,
>> it seems that I have to write my own routine?
>>
>>> Running for a Krylov method for tens of thousands of iterations is very
>>> rarely recommended.
>>>
>> GAMG and BCGS is the only ones that have actually worked for me so far, I
>> increased the max_it because it was not enough with the default one. With
>> the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
>> need more accuracy so I increased the tol to 10^-7 and that is when it
>> crashed after ~51000 iterations.
>>
>> @Barry
>>
>>> I'm guessing you mange your own time stepping (and nonlinear solver if
>>> there is one)?
>>>
>>
>> It is the default from the solver (attached the short code).
>>
>>> As Jed says it doesn't make sense to save "partial" solutions within the
>>> linear solver.  Just save it every several time-steps.
>>>
>>
>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>> such that the next time it is running it can pick up from where it crashed.
>>
>>    Also the code should not be "crashing" at seemingly long times (after
>>> hours) with a Segmentation fault. Send us the full error message and we'll
>>> see if there is some way we can help you fix this problem.
>>>
>> I have added this before, but seemingly they conclusion was that it is
>> memory error...although I am not sure how, but I have added the entire
>> error.
>>
>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>> wrote:
>>
>>>
>>>    I'm guessing you mange your own time stepping (and nonlinear solver
>>> if there is one)?
>>>
>>>    You can save the solution with a call to VecView() and then reload
>>> the solution with a VecLoad() but you need to manage any other restart data
>>> that you may need, like the value of the current time etc.
>>>
>>>    As Jed says it doesn't make sense to save "partial" solutions within
>>> the linear solver.  Just save it every several time-steps.
>>>
>>>    Barry
>>>
>>>    Also the code should not be "crashing" at seemingly long times (after
>>> hours) with a Segmentation fault. Send us the full error message and we'll
>>> see if there is some way we can help you fix this problem.
>>>
>>>
>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >
>>> > Is there a function/command line option to save the solution as it is
>>> solving (and read in the file from where it crashed and keep iterating from
>>> there perhaps)?
>>> > Had a seg fault and all the results until that point seems to have
>>> been lost.
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/b3a44572/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/b3a44572/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1-s2.0-S001046551300204X-main.pdf
Type: application/pdf
Size: 1766835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/b3a44572/attachment-0001.pdf>

From knepley at gmail.com  Tue Feb 19 07:40:12 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 08:40:12 -0500
Subject: [petsc-users] Including constrained dofs in a matrix/vector
 created with DMCreateXXXX()
In-Reply-To: <be667eba-cec1-27be-02bf-878f3090fb07@rice.edu>
References: <be667eba-cec1-27be-02bf-878f3090fb07@rice.edu>
Message-ID: <CAMYG4GmbhWj_Q9fFLf6ADY6ZSWvQJLyXKs5qTfB7h43K2T-X2A@mail.gmail.com>

On Tue, Feb 19, 2019 at 1:13 AM Jordan Wagner via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> Over the past few months, I have implemented dmplex/section in my
> preexisting finite element code. We already had our own implementations for
> things like the finite element space, boundary conditions, projections,
> etc, so I am mostly using dmplex/section at its lowest level; no DT/DS
> stuff for the moment. Everything is working fine at this level, and I am
> overall very happy with how it ties into the code that we already have.
>
> I am currently doing my own preallocation for the system Jacobian but am
> wanting to use DMCreateMatrix() instead, as it seems pointless to keep
> using my own crappy function when this one is just sitting there waiting.
> However, I dislike how I am currently implementing this and was hoping to
> get some suggestions or some advice. My problem is handling nicely the
> Dirichlet dofs. My code takes the approach of first assembling the entire
> Jacobian matrix and load vector as if no constraints are imposed. We then
> have a function that applies the essential conditions after assembly and
> extracts a submatrix and subvector similar to the way ex49
> <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex49.c.html>
> is doing.
>
> Since DMCreateMatrix() automatically partitions out the constraints
> specified in the section, I find myself having to create two nearly
> identical sections: one that has constraints and one that does not, and
> then I clone the DM and set the default section of each respectively. I can
> then use DMCreateMatrix() on the one dm/section with no constraints to
> preallocate the larger matrix that I am assembling. Then I use the
> dm/section with constraints to do all of my boundary condition and
> projection operations on the previously created matrix and vector.
> Essentially, I am using an unconstrained section only for allocation and a
> constrained section for most everything else.
>
> This process actually works fine for me, but its pretty ugly, and I am
> sure there is a better way. I am wondering if I am missing something that
> could keep me from having to go through this process of creating two
> sections that differ only in the constraints. It seems to me if there were
> an option for DMCreateMatrix()/DMCreateVector() to either include or not
> include the dofs, that would completely solve my problem. That way, I can
> use the same section for both creation and allocation.
>
> So, the main question I have is just whether there is a function or option
> that I am missing like DMCreateXXXX() but with the ability to retain the
> constrained rows and columns rather than compressing them out.
>
> I do not completely understand the above, but here is how Section works. A
_local_ Section contains all dofs and is intended for assembly using local
Vecs. To communicate with the solver, you make a _global_ Section using
PetscSectionCreateGlobalSection(). This has an option to include or exclude
constraints. For example, I make a global Section with constraints when I
output for visualization. Maybe you can use that to do what you want.

What I do not understand is why you do not just use a global Section
without constraints to assemble your matrix, rather than eliminating the
rows/columns later. The global Section gives back negative indices for
unknown that are constrained or not owned. These are ignored by
MatSetValues() and VecSetValues(). This is how the default Plex assembly
makes operators without constrained variables.

  Thanks,

     Matt

> Thanks a lot, as always.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/e4828352/attachment.html>

From knepley at gmail.com  Tue Feb 19 07:49:38 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 08:49:38 -0500
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C4fTAOfmp7Z5jYVOqR-hdesdks3=BdQPtp+DSU7S852yA@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
	<CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
	<CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>
	<CAE3k7C4fTAOfmp7Z5jYVOqR-hdesdks3=BdQPtp+DSU7S852yA@mail.gmail.com>
Message-ID: <CAMYG4GmnZC3j+zV8vncSbRYRSuvtViw9JgDcJdk5kpEbmH0P+g@mail.gmail.com>

On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempohoper at gmail.com> wrote:

> Matthew maybe indeed I was not clear and wrong in the wording. What I
> meant is the matrix is undefined, but since it is a finite element method
> (using second order elements). The pattern is like this:
>

Okay, the paper says that the formulation of Maxwell in frequency space is
"well conditioned", and in the paper they solve using
BiCG/Jacobi:

  -ksp_type bicg -pc_type jacobi

Since you are taking thousands of iterates:

  a) They are mistaken and the formulation is not well-conditioned

  b) You have a bug in the discretization

  c) You are outside the parameter range/boundary conditions/forcing they
were talking about for conditioning

Just to get something solving so you can check the discretization, I would

  a) Use SuperLU or MUMPS, -pc_type lu

  b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu

Then finally

  c) For very big problems, I suspect that PCPATCHcould be made to work
with the right choices, but this is a research problem

  Thanks,

     Matt


> [image: image.png]
> We are indeed solving Maxwell's equations. I have added the paper.
>
> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> It is a finite-element problem of an RF antenna dielectric interaction
>>> where all the non-zero elements are on the diagonal of the sparse matrix
>>> (if that is relevant).
>>>
>>
>> I am unsure whether this is what you mean. If you matrix is diagonal, you
>> can invert it exactly in a single step so you
>> do not need a Krylov solver. What equations are you solving? Maxwell?
>> Electrostatic? What finite element are you using?
>>   Thanks,
>>
>>     Matt
>>
>>
>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor,
>>> it seems that I have to write my own routine?
>>>
>>>> Running for a Krylov method for tens of thousands of iterations is very
>>>> rarely recommended.
>>>>
>>> GAMG and BCGS is the only ones that have actually worked for me so far,
>>> I increased the max_it because it was not enough with the default one. With
>>> the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
>>> need more accuracy so I increased the tol to 10^-7 and that is when it
>>> crashed after ~51000 iterations.
>>>
>>> @Barry
>>>
>>>> I'm guessing you mange your own time stepping (and nonlinear solver if
>>>> there is one)?
>>>>
>>>
>>> It is the default from the solver (attached the short code).
>>>
>>>> As Jed says it doesn't make sense to save "partial" solutions within
>>>> the linear solver.  Just save it every several time-steps.
>>>>
>>>
>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>>> such that the next time it is running it can pick up from where it crashed.
>>>
>>>    Also the code should not be "crashing" at seemingly long times (after
>>>> hours) with a Segmentation fault. Send us the full error message and we'll
>>>> see if there is some way we can help you fix this problem.
>>>>
>>> I have added this before, but seemingly they conclusion was that it is
>>> memory error...although I am not sure how, but I have added the entire
>>> error.
>>>
>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>>> wrote:
>>>
>>>>
>>>>    I'm guessing you mange your own time stepping (and nonlinear solver
>>>> if there is one)?
>>>>
>>>>    You can save the solution with a call to VecView() and then reload
>>>> the solution with a VecLoad() but you need to manage any other restart data
>>>> that you may need, like the value of the current time etc.
>>>>
>>>>    As Jed says it doesn't make sense to save "partial" solutions within
>>>> the linear solver.  Just save it every several time-steps.
>>>>
>>>>    Barry
>>>>
>>>>    Also the code should not be "crashing" at seemingly long times
>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>> we'll see if there is some way we can help you fix this problem.
>>>>
>>>>
>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>> >
>>>> > Is there a function/command line option to save the solution as it is
>>>> solving (and read in the file from where it crashed and keep iterating from
>>>> there perhaps)?
>>>> > Had a seg fault and all the results until that point seems to have
>>>> been lost.
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/5e3b6c50/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/5e3b6c50/attachment-0001.png>

From pietro.benedusi at usi.ch  Tue Feb 19 09:06:56 2019
From: pietro.benedusi at usi.ch (Pietro Benedusi)
Date: Tue, 19 Feb 2019 15:06:56 +0000
Subject: [petsc-users] PCMG: level dependent smoothing steps
Message-ID: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>

Dear PETSc team,

Is there a way to set the number of smoothing steps done inside a specific multigrid level l ?
Something like:

PCMGSetNumberSmoothDown<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PCMG/PCMGSetNumberSmoothDown.html#PCMGSetNumberSmoothDown>(PC<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PC/PC.html#PC> pc,PetscInt<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> n, PetscInt<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> l)

In this way I could, for example set 3 smoothing steps on level 1 and 7 smoothing steps on level 2.

Thank you!
Pietro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/f163bff2/attachment.html>

From patrick.sanan at gmail.com  Tue Feb 19 09:19:02 2019
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Tue, 19 Feb 2019 16:19:02 +0100
Subject: [petsc-users] PCMG: level dependent smoothing steps
In-Reply-To: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>
References: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>
Message-ID: <CA+z91Tf6CbJX=Q2HH+3gOfr105KjgUfeyP1wrVxHFjeGvje+Fw@mail.gmail.com>

Will it work for you to use

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGGetSmoother.html

to pull out the KSP for the level you're interested in (Note link to
PCMGGetSmootherDown() if you want to control up and down smoothers
separately) and then use

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetTolerances.html

to set the maximum iterations?

Am Di., 19. Feb. 2019 um 16:07 Uhr schrieb Pietro Benedusi via petsc-users <
petsc-users at mcs.anl.gov>:

> Dear PETSc team,
>
> Is there a way to set the number of smoothing steps done inside a specific
> multigrid level l ?
> Something like:
>
> PCMGSetNumberSmoothDown <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PCMG/PCMGSetNumberSmoothDown.html#PCMGSetNumberSmoothDown>(PC <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PC/PC.html#PC> pc,PetscInt <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> n, PetscInt <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> l)
>
>
> In this way I could, for example set 3 smoothing steps on level 1 and 7
> smoothing steps on level 2.
>
> Thank you!
> Pietro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/7cfc4093/attachment.html>

From knepley at gmail.com  Tue Feb 19 09:27:28 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 10:27:28 -0500
Subject: [petsc-users] PCMG: level dependent smoothing steps
In-Reply-To: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>
References: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>
Message-ID: <CAMYG4GmU_ggjV+HKzRa5j15Q1Q6R25_A08B71ec219ZjfP8Vgg@mail.gmail.com>

On Tue, Feb 19, 2019 at 10:07 AM Pietro Benedusi via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear PETSc team,
>
> Is there a way to set the number of smoothing steps done inside a specific
> multigrid level l ?
> Something like:
>
> PCMGSetNumberSmoothDown <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PCMG/PCMGSetNumberSmoothDown.html#PCMGSetNumberSmoothDown>(PC <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PC/PC.html#PC> pc,PetscInt <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> n, PetscInt <https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> l)
>
>
> In this way I could, for example set 3 smoothing steps on level 1 and 7
> smoothing steps on level 2.
>

-mg_level_1_ksp_max_it 3 -mg_level_2_ksp_mat_it 7

  Thanks,

     Matt


> Thank you!
> Pietro
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/d68e0ac2/attachment.html>

From pietro.benedusi at usi.ch  Tue Feb 19 09:28:44 2019
From: pietro.benedusi at usi.ch (Pietro Benedusi)
Date: Tue, 19 Feb 2019 15:28:44 +0000
Subject: [petsc-users] PCMG: level dependent smoothing steps
In-Reply-To: <CA+z91Tf6CbJX=Q2HH+3gOfr105KjgUfeyP1wrVxHFjeGvje+Fw@mail.gmail.com>
References: <61B04048-FB31-4D20-A0EA-5BB82700290E@usi.ch>
	<CA+z91Tf6CbJX=Q2HH+3gOfr105KjgUfeyP1wrVxHFjeGvje+Fw@mail.gmail.com>
Message-ID: <22B54A44-3D49-42BC-A7C7-7F1EADA9A79A@usi.ch>

Hi Patrick,

It worked, thank you!

Cheers from Lugano,
Pietro

Il giorno 19-feb-2019, alle ore 16:19, Patrick Sanan <patrick.sanan at gmail.com<mailto:patrick.sanan at gmail.com>> ha scritto:

Will it work for you to use

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMGGetSmoother.html

to pull out the KSP for the level you're interested in (Note link to PCMGGetSmootherDown() if you want to control up and down smoothers separately) and then use

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetTolerances.html

to set the maximum iterations?

Am Di., 19. Feb. 2019 um 16:07 Uhr schrieb Pietro Benedusi via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>:
Dear PETSc team,

Is there a way to set the number of smoothing steps done inside a specific multigrid level l ?
Something like:

PCMGSetNumberSmoothDown<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PCMG/PCMGSetNumberSmoothDown.html#PCMGSetNumberSmoothDown>(PC<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/PC/PC.html#PC> pc,PetscInt<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> n, PetscInt<https://www.mcs.anl.gov/petsc/petsc-3.8/docs/manualpages/Sys/PetscInt.html#PetscInt> l)

In this way I could, for example set 3 smoothing steps on level 1 and 7 smoothing steps on level 2.

Thank you!
Pietro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/1c1e46dd/attachment-0001.html>

From knepley at gmail.com  Tue Feb 19 09:48:21 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 10:48:21 -0500
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
Message-ID: <CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>

You basically need the inverse of MatCreateSubmatrices(). I do not think we
have that right now, but it could probably be done without too much trouble
by looking at that code.

  Thanks,

     Matt

On Tue, Feb 19, 2019 at 6:15 AM Marius Buerkle via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi !
>
> Is there some way to combine MatCompositeMerge
> with MatCreateRedundantMatrix? I basically want to create copies of a
> matrix from PETSC_COMM_WORLD to subcommunicators, do some work on each
> subcommunicator and than gather the results back to PETSC_COMM_WORLD,
> namely  I want to sum the  the invidual matrices from the subcommunicatos
> component wise and get the resulting matrix on PETSC_COMM_WORLD. Is this
> somehow possible without going through all the hassle of using MPI
> directly?
>
> marius
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/8c773fcb/attachment.html>

From wence at gmx.li  Tue Feb 19 09:59:53 2019
From: wence at gmx.li (Lawrence Mitchell)
Date: Tue, 19 Feb 2019 15:59:53 +0000
Subject: [petsc-users] Identifying matching points in differently
 distributed DMPlexs
Message-ID: <A77C8520-D12C-4E75-9D2E-54AA3821B602@gmx.li>

Dear petsc-users,

I have two (different) distributions of the (topologically) same DMPlex object (DM_a and DM_b).

I would like to identify the map from points(DM_a) to points(DM_b) such that I can transfer fields between the two.

Does such a facility exist?

Lawrence

From knepley at gmail.com  Tue Feb 19 15:30:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 16:30:16 -0500
Subject: [petsc-users] Identifying matching points in differently
 distributed DMPlexs
In-Reply-To: <A77C8520-D12C-4E75-9D2E-54AA3821B602@gmx.li>
References: <A77C8520-D12C-4E75-9D2E-54AA3821B602@gmx.li>
Message-ID: <CAMYG4G=p_6+SuORjFosQ3HuUG7FTK5sOEEMSOQXbJN8_SB23zA@mail.gmail.com>

On Tue, Feb 19, 2019 at 11:00 AM Lawrence Mitchell via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear petsc-users,
>
> I have two (different) distributions of the (topologically) same DMPlex
> object (DM_a and DM_b).
>
> I would like to identify the map from points(DM_a) to points(DM_b) such
> that I can transfer fields between the two.
>
> Does such a facility exist?
>

Case #1: You made DM_b from DM_a

  Then you are fine. The PetscSF you get from Distribute() is the map.

Case #2: They are completely unconnected

  This is hard. I wrote code to do it at least twice, but I keep throwing
it away when I rewrite things because
it is so ugly. Hopefully you do not need this.

  Thanks,

     Matt


> Lawrence



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/7f31305d/attachment.html>

From mbuerkle at web.de  Tue Feb 19 18:12:15 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Wed, 20 Feb 2019 01:12:15 +0100
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
Message-ID: <trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/a3930fdb/attachment.html>

From knepley at gmail.com  Tue Feb 19 19:09:35 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 20:09:35 -0500
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
	<trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
Message-ID: <CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>

On Tue, Feb 19, 2019 at 7:12 PM Marius Buerkle <mbuerkle at web.de> wrote:

> I see. This would work if the matrices are on different subcommumicators ?
> Is it possible to add this functionality ?
>

Hmm, no. That is specialized to serial matrices. You need the inverse of
MatCreateSubMatricesMPI().

  Thanks,

     Matt


> marius
>
>
> You basically need the inverse of MatCreateSubmatrices(). I do not think
> we have that right now, but it could probably be done without too much
> trouble by looking at that code.
>
>   Thanks,
>
>      Matt
>
> On Tue, Feb 19, 2019 at 6:15 AM Marius Buerkle via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi !
>>
>> Is there some way to combine MatCompositeMerge
>> with MatCreateRedundantMatrix? I basically want to create copies of a
>> matrix from PETSC_COMM_WORLD to subcommunicators, do some work on each
>> subcommunicator and than gather the results back to PETSC_COMM_WORLD,
>> namely  I want to sum the  the invidual matrices from the subcommunicatos
>> component wise and get the resulting matrix on PETSC_COMM_WORLD. Is this
>> somehow possible without going through all the hassle of using MPI
>> directly?
>>
>> marius
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/9fe6dcdc/attachment.html>

From mbuerkle at web.de  Tue Feb 19 19:42:13 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Wed, 20 Feb 2019 02:42:13 +0100
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
	<trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
	<CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>
Message-ID: <trinity-3a2ccd9b-25fa-4695-a866-6a07995cefae-1550626933255@3c-app-webde-bs38>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/135f56ad/attachment.html>

From knepley at gmail.com  Tue Feb 19 19:44:26 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 Feb 2019 20:44:26 -0500
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <trinity-3a2ccd9b-25fa-4695-a866-6a07995cefae-1550626933255@3c-app-webde-bs38>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
	<trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
	<CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>
	<trinity-3a2ccd9b-25fa-4695-a866-6a07995cefae-1550626933255@3c-app-webde-bs38>
Message-ID: <CAMYG4Gkr=jfY=tqqAukEBtmMMvtKuEwhLfBvAaw_j54Q41o5Kg@mail.gmail.com>

On Tue, Feb 19, 2019 at 8:42 PM Marius Buerkle <mbuerkle at web.de> wrote:

> ok, so it seems there is no straight forward way to transfer data between
> PETSc matrices on different subcomms. Probably doing it by "hand"
> extracting the matricies on the subcomms create a MPI_INTERCOMM transfering
> the data to PETSC_COMM_WORLD and assembling them in a new PETSc matrix
> would be possible, right?
>

That sounds too complicated. Why not just reverse
MPICreateSubMatricesMPI()? Meaning make it collective on the whole big
communicator, so that you can swap out all the subcommunicator for the
aggregation call, just like we do in that function.
Then its really just a matter of reversing the communication call.

   Matt

>
> On Tue, Feb 19, 2019 at 7:12 PM Marius Buerkle <mbuerkle at web.de> wrote:
>
>> I see. This would work if the matrices are on different subcommumicators
>> ? Is it possible to add this functionality ?
>>
>
> Hmm, no. That is specialized to serial matrices. You need the inverse of
> MatCreateSubMatricesMPI().
>
>   Thanks,
>
>      Matt
>
>
>> marius
>>
>>
>> You basically need the inverse of MatCreateSubmatrices(). I do not think
>> we have that right now, but it could probably be done without too much
>> trouble by looking at that code.
>>
>>   Thanks,
>>
>>      Matt
>>
>> On Tue, Feb 19, 2019 at 6:15 AM Marius Buerkle via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi !
>>>
>>> Is there some way to combine MatCompositeMerge
>>> with MatCreateRedundantMatrix? I basically want to create copies of a
>>> matrix from PETSC_COMM_WORLD to subcommunicators, do some work on each
>>> subcommunicator and than gather the results back to PETSC_COMM_WORLD,
>>> namely  I want to sum the  the invidual matrices from the subcommunicatos
>>> component wise and get the resulting matrix on PETSC_COMM_WORLD. Is this
>>> somehow possible without going through all the hassle of using MPI
>>> directly?
>>>
>>> marius
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190219/369eff13/attachment-0001.html>

From tempohoper at gmail.com  Wed Feb 20 03:43:18 2019
From: tempohoper at gmail.com (Sal Am)
Date: Wed, 20 Feb 2019 09:43:18 +0000
Subject: [petsc-users] saving results
In-Reply-To: <CAMYG4GmnZC3j+zV8vncSbRYRSuvtViw9JgDcJdk5kpEbmH0P+g@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
	<CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
	<CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>
	<CAE3k7C4fTAOfmp7Z5jYVOqR-hdesdks3=BdQPtp+DSU7S852yA@mail.gmail.com>
	<CAMYG4GmnZC3j+zV8vncSbRYRSuvtViw9JgDcJdk5kpEbmH0P+g@mail.gmail.com>
Message-ID: <CAE3k7C4AHLGjVHAyAAHOuHYzyk5BvBZkRbqfHvnFp4xv6mnVrw@mail.gmail.com>

Hi Matthew you were right,

The matrix I have is very ill conditioned and my supervisor gave it for
testing purposes. Having said that, I was able to solve it previously
however, for some reason it said convergence reached at e-3 even though the
default convergence tol is e-5 which is why I put rel_tol to e-7 and it
crashed.

Just to get something solving so you can check the discretization, I would
>
>   a) Use SuperLU or MUMPS, -pc_type lu
>
>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>
> Then finally
>
>   c) For very big problems, I suspect that PCPATCHcould be made to work
> with the right choices, but this is a research problem
>

That is very interesting, I have tried a) before and it does not work (lack
of memory presumably). I will try to do b), but I am very interested in c).
the final BOSS fight so to say is the 30million finite element model of the
JET A2 Antenna so it would be great if I can utilise PETSc solving it.
(Perhaps I should open another ticket for that later on)

But to stay on topic I have only used VecView to actually view the solution
*after* it was done solving. Is there an example on how people actually
utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints
*during* iterations so we can pick up wherever it crashed?

Thanks

On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempohoper at gmail.com> wrote:
>
>> Matthew maybe indeed I was not clear and wrong in the wording. What I
>> meant is the matrix is undefined, but since it is a finite element method
>> (using second order elements). The pattern is like this:
>>
>
> Okay, the paper says that the formulation of Maxwell in frequency space is
> "well conditioned", and in the paper they solve using
> BiCG/Jacobi:
>
>   -ksp_type bicg -pc_type jacobi
>
> Since you are taking thousands of iterates:
>
>   a) They are mistaken and the formulation is not well-conditioned
>
>   b) You have a bug in the discretization
>
>   c) You are outside the parameter range/boundary conditions/forcing they
> were talking about for conditioning
>
> Just to get something solving so you can check the discretization, I would
>
>   a) Use SuperLU or MUMPS, -pc_type lu
>
>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>
> Then finally
>
>   c) For very big problems, I suspect that PCPATCHcould be made to work
> with the right choices, but this is a research problem
>
>   Thanks,
>
>      Matt
>
>
>> [image: image.png]
>> We are indeed solving Maxwell's equations. I have added the paper.
>>
>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> It is a finite-element problem of an RF antenna dielectric interaction
>>>> where all the non-zero elements are on the diagonal of the sparse matrix
>>>> (if that is relevant).
>>>>
>>>
>>> I am unsure whether this is what you mean. If you matrix is diagonal,
>>> you can invert it exactly in a single step so you
>>> do not need a Krylov solver. What equations are you solving? Maxwell?
>>> Electrostatic? What finite element are you using?
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor,
>>>> it seems that I have to write my own routine?
>>>>
>>>>> Running for a Krylov method for tens of thousands of iterations is
>>>>> very rarely recommended.
>>>>>
>>>> GAMG and BCGS is the only ones that have actually worked for me so far,
>>>> I increased the max_it because it was not enough with the default one. With
>>>> the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
>>>> need more accuracy so I increased the tol to 10^-7 and that is when it
>>>> crashed after ~51000 iterations.
>>>>
>>>> @Barry
>>>>
>>>>> I'm guessing you mange your own time stepping (and nonlinear solver if
>>>>> there is one)?
>>>>>
>>>>
>>>> It is the default from the solver (attached the short code).
>>>>
>>>>> As Jed says it doesn't make sense to save "partial" solutions within
>>>>> the linear solver.  Just save it every several time-steps.
>>>>>
>>>>
>>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>>>> such that the next time it is running it can pick up from where it crashed.
>>>>
>>>>    Also the code should not be "crashing" at seemingly long times
>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>
>>>> I have added this before, but seemingly they conclusion was that it is
>>>> memory error...although I am not sure how, but I have added the entire
>>>> error.
>>>>
>>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>>
>>>>>    I'm guessing you mange your own time stepping (and nonlinear solver
>>>>> if there is one)?
>>>>>
>>>>>    You can save the solution with a call to VecView() and then reload
>>>>> the solution with a VecLoad() but you need to manage any other restart data
>>>>> that you may need, like the value of the current time etc.
>>>>>
>>>>>    As Jed says it doesn't make sense to save "partial" solutions
>>>>> within the linear solver.  Just save it every several time-steps.
>>>>>
>>>>>    Barry
>>>>>
>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>
>>>>>
>>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> >
>>>>> > Is there a function/command line option to save the solution as it
>>>>> is solving (and read in the file from where it crashed and keep iterating
>>>>> from there perhaps)?
>>>>> > Had a seg fault and all the results until that point seems to have
>>>>> been lost.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/405f75f5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/405f75f5/attachment-0001.png>

From knepley at gmail.com  Wed Feb 20 05:29:48 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 20 Feb 2019 06:29:48 -0500
Subject: [petsc-users] saving results
In-Reply-To: <CAE3k7C4AHLGjVHAyAAHOuHYzyk5BvBZkRbqfHvnFp4xv6mnVrw@mail.gmail.com>
References: <CAE3k7C5vaVAOFrHq5forNnx0hkzVeySfMtdrzg0YptfcjQqt=g@mail.gmail.com>
	<FC99A601-2AF6-49A8-BCCF-FC293ECD575D@anl.gov>
	<CAE3k7C4exjypjn4Wj0-TtGSSGtU+Ume4KLyRYtk4sXrR28d56Q@mail.gmail.com>
	<CAMYG4GmNCP937Kj-YMUJgCO5qQF5X4c8fuR504O9qbSVgxKLyQ@mail.gmail.com>
	<CAE3k7C4fTAOfmp7Z5jYVOqR-hdesdks3=BdQPtp+DSU7S852yA@mail.gmail.com>
	<CAMYG4GmnZC3j+zV8vncSbRYRSuvtViw9JgDcJdk5kpEbmH0P+g@mail.gmail.com>
	<CAE3k7C4AHLGjVHAyAAHOuHYzyk5BvBZkRbqfHvnFp4xv6mnVrw@mail.gmail.com>
Message-ID: <CAMYG4G=u6JQ=uoiNnVDaPuupiKTt6ymLh7kd5iFwT0VAWdzaiw@mail.gmail.com>

On Wed, Feb 20, 2019 at 4:43 AM Sal Am <tempohoper at gmail.com> wrote:

> Hi Matthew you were right,
>
> The matrix I have is very ill conditioned and my supervisor gave it for
> testing purposes. Having said that, I was able to solve it previously
> however, for some reason it said convergence reached at e-3 even though the
> default convergence tol is e-5 which is why I put rel_tol to e-7 and it
> crashed.
>
> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>
> That is very interesting, I have tried a) before and it does not work
> (lack of memory presumably). I will try to do b), but I am very interested
> in c). the final BOSS fight so to say is the 30million finite element model
> of the JET A2 Antenna so it would be great if I can utilise PETSc solving
> it. (Perhaps I should open another ticket for that later on)
>

Yep. The reason I thought it might work is that there is theory for it
working on positive semi-definite matrices (matrices which are
almost nice but can have a large null space). The idea is to capture the
local nullspace in each patch. There is an excellent writeup
of the idea, with all appropriate references, here:
https://arxiv.org/abs/1810.03315

But to stay on topic I have only used VecView to actually view the solution
> *after* it was done solving. Is there an example on how people actually
> utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints
> *during* iterations so we can pick up wherever it crashed?
>

Sure. Here we set a custom monitor


https://bitbucket.org/petsc/petsc/src/6f3214441ec7b7dca9916d2bd112d01eef3e7185/src/ksp/ksp/examples/tutorials/ex9.c#lines-111

Just stick your VecView in there. In fact, I would use VecViewFromOptions()
so you can decide what viewer to use on the command line.

  Thanks,

     Matt


> Thanks
>
> On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempohoper at gmail.com> wrote:
>>
>>> Matthew maybe indeed I was not clear and wrong in the wording. What I
>>> meant is the matrix is undefined, but since it is a finite element method
>>> (using second order elements). The pattern is like this:
>>>
>>
>> Okay, the paper says that the formulation of Maxwell in frequency space
>> is "well conditioned", and in the paper they solve using
>> BiCG/Jacobi:
>>
>>   -ksp_type bicg -pc_type jacobi
>>
>> Since you are taking thousands of iterates:
>>
>>   a) They are mistaken and the formulation is not well-conditioned
>>
>>   b) You have a bug in the discretization
>>
>>   c) You are outside the parameter range/boundary conditions/forcing they
>> were talking about for conditioning
>>
>> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> [image: image.png]
>>> We are indeed solving Maxwell's equations. I have added the paper.
>>>
>>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> It is a finite-element problem of an RF antenna dielectric interaction
>>>>> where all the non-zero elements are on the diagonal of the sparse matrix
>>>>> (if that is relevant).
>>>>>
>>>>
>>>> I am unsure whether this is what you mean. If you matrix is diagonal,
>>>> you can invert it exactly in a single step so you
>>>> do not need a Krylov solver. What equations are you solving? Maxwell?
>>>> Electrostatic? What finite element are you using?
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the
>>>>> KSPMonitor, it seems that I have to write my own routine?
>>>>>
>>>>>> Running for a Krylov method for tens of thousands of iterations is
>>>>>> very rarely recommended.
>>>>>>
>>>>> GAMG and BCGS is the only ones that have actually worked for me so
>>>>> far, I increased the max_it because it was not enough with the default one.
>>>>> With the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but
>>>>> I need more accuracy so I increased the tol to 10^-7 and that is when it
>>>>> crashed after ~51000 iterations.
>>>>>
>>>>> @Barry
>>>>>
>>>>>> I'm guessing you mange your own time stepping (and nonlinear solver
>>>>>> if there is one)?
>>>>>>
>>>>>
>>>>> It is the default from the solver (attached the short code).
>>>>>
>>>>>> As Jed says it doesn't make sense to save "partial" solutions within
>>>>>> the linear solver.  Just save it every several time-steps.
>>>>>>
>>>>>
>>>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>>>>> such that the next time it is running it can pick up from where it crashed.
>>>>>
>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>>
>>>>> I have added this before, but seemingly they conclusion was that it is
>>>>> memory error...although I am not sure how, but I have added the entire
>>>>> error.
>>>>>
>>>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>    I'm guessing you mange your own time stepping (and nonlinear
>>>>>> solver if there is one)?
>>>>>>
>>>>>>    You can save the solution with a call to VecView() and then reload
>>>>>> the solution with a VecLoad() but you need to manage any other restart data
>>>>>> that you may need, like the value of the current time etc.
>>>>>>
>>>>>>    As Jed says it doesn't make sense to save "partial" solutions
>>>>>> within the linear solver.  Just save it every several time-steps.
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>>
>>>>>>
>>>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>> >
>>>>>> > Is there a function/command line option to save the solution as it
>>>>>> is solving (and read in the file from where it crashed and keep iterating
>>>>>> from there perhaps)?
>>>>>> > Had a seg fault and all the results until that point seems to have
>>>>>> been lost.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/b2ccc135/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/b2ccc135/attachment-0001.png>

From myriam.peyrounette at idris.fr  Wed Feb 20 08:08:37 2019
From: myriam.peyrounette at idris.fr (Myriam Peyrounette)
Date: Wed, 20 Feb 2019 15:08:37 +0100
Subject: [petsc-users] PCMGSetGalerkin() new inputs
Message-ID: <55dc11b9-3751-9e5a-56f1-e32ed05f5216@idris.fr>

Hi,

I am currently comparing two codes based on PETSc. The first one uses
PETSC 3.6.4 and the other one PETSc 3.10.2.

I am having a look at the use of the function PCMGSetGalerkin(). With
PETSc 3.6, the input is a boolean, while it is either
PC_MG_GALERKIN_MAT, PC_MG_GALERKIN_PMAT or PC_MG_GALERKIN_BOTH with PETSc 3.10.

Which one of these new inputs should I use to have the same
configuration as with PETSc 3.6?

Thanks in advance,

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/de54fc02/attachment.p7s>

From jed at jedbrown.org  Wed Feb 20 10:16:29 2019
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 20 Feb 2019 09:16:29 -0700
Subject: [petsc-users] PCMGSetGalerkin() new inputs
In-Reply-To: <55dc11b9-3751-9e5a-56f1-e32ed05f5216@idris.fr>
References: <55dc11b9-3751-9e5a-56f1-e32ed05f5216@idris.fr>
Message-ID: <8736oicu5e.fsf@jedbrown.org>

This wasn't explained well in the commit message.  The old code used the
Galerkin procedure on the "Pmat" (preconditioning matrix; which may or
may not be the same as the Amat) and set the result as both Amat and
Pmat of the coarse grid.  The new code allows you to specify.  If your
Amat and Pmat are the same on the finest level, then BOTH will offer the
same behavior as before.

Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hi,
>
> I am currently comparing two codes based on PETSc. The first one uses
> PETSC 3.6.4 and the other one PETSc 3.10.2.
>
> I am having a look at the use of the function PCMGSetGalerkin(). With
> PETSc 3.6, the input is a boolean, while it is either
> PC_MG_GALERKIN_MAT, PC_MG_GALERKIN_PMAT or PC_MG_GALERKIN_BOTH with PETSc 3.10.
>
> Which one of these new inputs should I use to have the same
> configuration as with PETSc 3.6?
>
> Thanks in advance,
>
> -- 
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --

From ellen.price at cfa.harvard.edu  Wed Feb 20 12:42:05 2019
From: ellen.price at cfa.harvard.edu (Ellen Price)
Date: Wed, 20 Feb 2019 13:42:05 -0500
Subject: [petsc-users] Using DMCOMPOSITE with TS
Message-ID: <CAOnA6qFX2vxMwM6xNxV_YBK1zjebZUPWsJZyu8wXetyzutFGdA@mail.gmail.com>

Hi fellow PETSc users!

I am attempting to use a DMCOMPOSITE alongside TS and have run into some
trouble. I'm attaching a MWE demonstrating the problem. The goal is to
combine a DMDA3d for spatial data and a DMREDUNDANT for non-spatial,
time-dependent fields. In the attached example, this additional field is
temperature.

The DMDA data is currently just temperature-dependent, and the temperature
is supposed to increase linearly. Unfortunately, no matter how I integrate
(explicitly, using Euler or RK, or implicitly, using backward Euler or
BDF), I get the wrong final temperature. The integration proceeds for 100
timesteps and then stops (verified by -ts_monitor). At a heating rate of 1,
and an initial temperature of 150, I should get a final temperature of 250
very easily. However, I get something closer to 200 (not exact, which makes
me think this isn't a simple missing-a-factor-of-2 error).

I'm currently using TSSetDM with the composite DM to let PETSc compute the
Jacobian. Having checked all the inputs and outputs as well as I can, this
is the only step that seems like it may be causing a problem; yet even
explicit integration doesn't work, and that shouldn't need the Jacobian at
all. I'm at a loss.

Run using:
./mwe -implicit yes -ts_type beuler -ts_monitor OR ./mwe -implicit no
-ts_type euler -ts_monitor

Thanks in advance for any help,
Ellen Price
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/7234eb35/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mwe.c
Type: text/x-csrc
Size: 6467 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/7234eb35/attachment.bin>

From j.wagner at rice.edu  Wed Feb 20 14:38:32 2019
From: j.wagner at rice.edu (Jordan Wagner)
Date: Wed, 20 Feb 2019 14:38:32 -0600
Subject: [petsc-users] Including constrained dofs in a matrix/vector
 created with DMCreateXXXX()
In-Reply-To: <CAMYG4GmbhWj_Q9fFLf6ADY6ZSWvQJLyXKs5qTfB7h43K2T-X2A@mail.gmail.com>
References: <be667eba-cec1-27be-02bf-878f3090fb07@rice.edu>
	<CAMYG4GmbhWj_Q9fFLf6ADY6ZSWvQJLyXKs5qTfB7h43K2T-X2A@mail.gmail.com>
Message-ID: <0304c38e-5124-2c8a-a1c2-ab16f9f51346@rice.edu>

On 2/19/19 7:40 AM, Matthew Knepley wrote:
> On Tue, Feb 19, 2019 at 1:13 AM Jordan Wagner via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Hi,
>
>     Over the past few months, I have implemented dmplex/section in my
>     preexisting finite element code. We already had our own
>     implementations for things like the finite element space, boundary
>     conditions, projections, etc, so I am mostly using dmplex/section
>     at its lowest level; no DT/DS stuff for the moment. Everything is
>     working fine at this level, and I am overall very happy with how
>     it ties into the code that we already have.
>
>     I am currently doing my own preallocation for the system Jacobian
>     but am wanting to use DMCreateMatrix() instead, as it seems
>     pointless to keep using my own crappy function when this one is
>     just sitting there waiting. However, I dislike how I am currently
>     implementing this and was hoping to get some suggestions or some
>     advice. My problem is handling nicely the Dirichlet dofs. My code
>     takes the approach of first assembling the entire Jacobian matrix
>     and load vector as if no constraints are imposed. We then have a
>     function that applies the essential conditions after assembly and
>     extracts a submatrix and subvector similar to the way ex49
>     <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex49.c.html>
>     is doing.
>
>     Since DMCreateMatrix() automatically partitions out the
>     constraints specified in the section, I find myself having to
>     create two nearly identical sections: one that has constraints and
>     one that does not, and then I clone the DM and set the default
>     section of each respectively. I can then use DMCreateMatrix() on
>     the one dm/section with no constraints to preallocate the larger
>     matrix that I am assembling. Then I use the dm/section with
>     constraints to do all of my boundary condition and projection
>     operations on the previously created matrix and vector.
>     Essentially, I am using an unconstrained section only for
>     allocation and a constrained section for most everything else.
>
>     This process actually works fine for me, but its pretty ugly, and
>     I am sure there is a better way. I am wondering if I am missing
>     something that could keep me from having to go through this
>     process of creating two sections that differ only in the
>     constraints. It seems to me if there were an option for
>     DMCreateMatrix()/DMCreateVector() to either include or not include
>     the dofs, that would completely solve my problem. That way, I can
>     use the same section for both creation and allocation.
>
>     So, the main question I have is just whether there is a function
>     or option that I am missing like DMCreateXXXX() but with the
>     ability to retain the constrained rows and columns rather than
>     compressing them out.
>
> I do not completely understand the above, but here is how Section 
> works. A _local_ Section contains all dofs and is intended for 
> assembly using local Vecs. To communicate with the solver, you make a 
> _global_ Section using PetscSectionCreateGlobalSection(). This has an 
> option to include or exclude constraints. For example, I make a global 
> Section with constraints when I output for visualization. Maybe you 
> can use that to do what you want.
>
Thanks for the prompt reply and sorry for the unclear initial question. 
I think this helps in my understanding of how this tool was intended to 
be used.

> What I do not understand is why you do not just use a global Section 
> without constraints to assemble your matrix, rather than eliminating 
> the rows/columns later. The global Section gives back negative indices 
> for unknown that are constrained or not owned. These are ignored by 
> MatSetValues() and VecSetValues(). This is how the default Plex 
> assembly makes operators without constrained variables.
>
I have rarely used a global section directly because I have never been 
able to? make sense of exactly what it was giving me. For instance as 
you mentioned (and I have seen written elsewhere), the global section 
should give negative indices for unowned or constrained dofs. This seems 
to work fine for me with the unowned points---I clearly get a negative 
size and offset from the section on these points. But my problem is for 
points that have constraints. I don't see where I would get negative 
indices from. The sizes and offsets for these points are not negative.? 
Of course, from the constraint size and constraint indices on these 
points, I can manually find out which should be negative and go from 
there. However, that leads me to the next problem I am having with 
global section...

For some reason, when I am creating the global section from the local 
one, my constrained component indices are getting messed up. The sizes 
are right but the order seems to be random and/or nonsensical. For 
instance, here is a snippet from my code being run in serial

Local section:

 ? (17615) dim? 2 offset 794 constrained 0
 ? (17616) dim? 2 offset 796 constrained 0 1
 ? (17617) dim? 2 offset 798 constrained 0 1
 ? (17618) dim? 2 offset 800 constrained 0 1
 ? (17619) dim? 2 offset 802 constrained 0 1
 ? (17620) dim? 2 offset 804 constrained 0 1
 ? (17621) dim? 2 offset 806 constrained 0 1

Global section:

 ? (17615) dim? 2 offset? 80 constrained 1
 ? (17616) dim? 2 offset? 81 constrained 0 0
 ? (17617) dim? 2 offset? 81 constrained 0 0
 ? (17618) dim? 2 offset? 81 constrained 0 0
 ? (17619) dim? 2 offset? 81 constrained 0 0
 ? (17620) dim? 2 offset? 81 constrained 0 0
 ? (17621) dim? 2 offset? 81 constrained 0 1

So using the global section, I am unable to even manually tell which 
components are constrained. Is this normal for the global section or is 
it something going wrong in its creation? or am I completely 
misunderstanding something?

Thanks very much.


> ? Thanks,
>
> ? ? ?Matt
>
>     Thanks a lot, as always.
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/b238b4d7/attachment.html>

From knepley at gmail.com  Wed Feb 20 15:15:16 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 20 Feb 2019 16:15:16 -0500
Subject: [petsc-users] Including constrained dofs in a matrix/vector
 created with DMCreateXXXX()
In-Reply-To: <0304c38e-5124-2c8a-a1c2-ab16f9f51346@rice.edu>
References: <be667eba-cec1-27be-02bf-878f3090fb07@rice.edu>
	<CAMYG4GmbhWj_Q9fFLf6ADY6ZSWvQJLyXKs5qTfB7h43K2T-X2A@mail.gmail.com>
	<0304c38e-5124-2c8a-a1c2-ab16f9f51346@rice.edu>
Message-ID: <CAMYG4GmyfSiKReWM75gvMvOLbremSVU=684WPOtkjLbv9cryAg@mail.gmail.com>

On Wed, Feb 20, 2019 at 3:38 PM Jordan Wagner <j.wagner at rice.edu> wrote:

> On 2/19/19 7:40 AM, Matthew Knepley wrote:
>
> On Tue, Feb 19, 2019 at 1:13 AM Jordan Wagner via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> Over the past few months, I have implemented dmplex/section in my
>> preexisting finite element code. We already had our own implementations for
>> things like the finite element space, boundary conditions, projections,
>> etc, so I am mostly using dmplex/section at its lowest level; no DT/DS
>> stuff for the moment. Everything is working fine at this level, and I am
>> overall very happy with how it ties into the code that we already have.
>>
>> I am currently doing my own preallocation for the system Jacobian but am
>> wanting to use DMCreateMatrix() instead, as it seems pointless to keep
>> using my own crappy function when this one is just sitting there waiting.
>> However, I dislike how I am currently implementing this and was hoping to
>> get some suggestions or some advice. My problem is handling nicely the
>> Dirichlet dofs. My code takes the approach of first assembling the entire
>> Jacobian matrix and load vector as if no constraints are imposed. We then
>> have a function that applies the essential conditions after assembly and
>> extracts a submatrix and subvector similar to the way ex49
>> <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex49.c.html>
>> is doing.
>>
>> Since DMCreateMatrix() automatically partitions out the constraints
>> specified in the section, I find myself having to create two nearly
>> identical sections: one that has constraints and one that does not, and
>> then I clone the DM and set the default section of each respectively. I can
>> then use DMCreateMatrix() on the one dm/section with no constraints to
>> preallocate the larger matrix that I am assembling. Then I use the
>> dm/section with constraints to do all of my boundary condition and
>> projection operations on the previously created matrix and vector.
>> Essentially, I am using an unconstrained section only for allocation and a
>> constrained section for most everything else.
>>
>> This process actually works fine for me, but its pretty ugly, and I am
>> sure there is a better way. I am wondering if I am missing something that
>> could keep me from having to go through this process of creating two
>> sections that differ only in the constraints. It seems to me if there were
>> an option for DMCreateMatrix()/DMCreateVector() to either include or not
>> include the dofs, that would completely solve my problem. That way, I can
>> use the same section for both creation and allocation.
>>
>> So, the main question I have is just whether there is a function or
>> option that I am missing like DMCreateXXXX() but with the ability to retain
>> the constrained rows and columns rather than compressing them out.
>>
> I do not completely understand the above, but here is how Section works. A
> _local_ Section contains all dofs and is intended for assembly using local
> Vecs. To communicate with the solver, you make a _global_ Section using
> PetscSectionCreateGlobalSection(). This has an option to include or exclude
> constraints. For example, I make a global Section with constraints when I
> output for visualization. Maybe you can use that to do what you want.
>
> Thanks for the prompt reply and sorry for the unclear initial question. I
> think this helps in my understanding of how this tool was intended to be
> used.
>
> What I do not understand is why you do not just use a global Section
> without constraints to assemble your matrix, rather than eliminating the
> rows/columns later. The global Section gives back negative indices for
> unknown that are constrained or not owned. These are ignored by
> MatSetValues() and VecSetValues(). This is how the default Plex assembly
> makes operators without constrained variables.
>
> I have rarely used a global section directly because I have never been
> able to  make sense of exactly what it was giving me. For instance as you
> mentioned (and I have seen written elsewhere), the global section should
> give negative indices for unowned or constrained dofs. This seems to work
> fine for me with the unowned points---I clearly get a negative size and
> offset from the section on these points. But my problem is for points that
> have constraints. I don't see where I would get negative indices from. The
> sizes and offsets for these points are not negative.  Of course, from the
> constraint size and constraint indices on these points, I can manually find
> out which should be negative and go from there. However, that leads me to
> the next problem I am having with global section...
>
> You are right. They do not give negative indices (I was just spacing out),
they do not give indices at all. The sizes in
the global section for those points are smaller than the local section.

> For some reason, when I am creating the global section from the local one,
> my constrained component indices are getting messed up. The sizes are right
> but the order seems to be random and/or nonsensical. For instance, here is
> a snippet from my code being run in serial
>
> That is a documentation failure. I did not copy over the constrained
indices into the global section because they
do not really mean anything there. In the local section, you can find the
dofs that get constrained, but in the global
section they are eliminated, so the numbering would not make sense.
Unfortunately, I did not think of a tricky way to
avoid printing them out. I guess I should set negative values and have the
printing check that. So as you say, the sizes
are right, and the constrained indices are meaningless.

  Thanks,

    Matt

> Local section:
>
>   (17615) dim  2 offset 794 constrained 0
>   (17616) dim  2 offset 796 constrained 0 1
>   (17617) dim  2 offset 798 constrained 0 1
>   (17618) dim  2 offset 800 constrained 0 1
>   (17619) dim  2 offset 802 constrained 0 1
>   (17620) dim  2 offset 804 constrained 0 1
>   (17621) dim  2 offset 806 constrained 0 1
>
> Global section:
>
>   (17615) dim  2 offset  80 constrained 1
>   (17616) dim  2 offset  81 constrained 0 0
>   (17617) dim  2 offset  81 constrained 0 0
>   (17618) dim  2 offset  81 constrained 0 0
>   (17619) dim  2 offset  81 constrained 0 0
>   (17620) dim  2 offset  81 constrained 0 0
>   (17621) dim  2 offset  81 constrained 0 1
>
> So using the global section, I am unable to even manually tell which
> components are constrained. Is this normal for the global section or is it
> something going wrong in its creation? or am I completely misunderstanding
> something?
>
> Thanks very much.
>
>
>   Thanks,
>
>      Matt
>
>> Thanks a lot, as always.
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/87f8005c/attachment-0001.html>

From mbuerkle at web.de  Wed Feb 20 17:57:43 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Thu, 21 Feb 2019 00:57:43 +0100
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <CAMYG4Gkr=jfY=tqqAukEBtmMMvtKuEwhLfBvAaw_j54Q41o5Kg@mail.gmail.com>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
	<trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
	<CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>
	<trinity-3a2ccd9b-25fa-4695-a866-6a07995cefae-1550626933255@3c-app-webde-bs38>
	<CAMYG4Gkr=jfY=tqqAukEBtmMMvtKuEwhLfBvAaw_j54Q41o5Kg@mail.gmail.com>
Message-ID: <trinity-4ea6b41e-2343-4b4e-9dc3-e21eb1c83c54-1550707063864@3c-app-webde-bap44>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/97e1b4a2/attachment.html>

From knepley at gmail.com  Wed Feb 20 18:31:44 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 20 Feb 2019 19:31:44 -0500
Subject: [petsc-users] MatCompositeMerge + MatCreateRedundantMatrix
In-Reply-To: <trinity-4ea6b41e-2343-4b4e-9dc3-e21eb1c83c54-1550707063864@3c-app-webde-bap44>
References: <trinity-e51ed7c5-344f-41cb-b493-4033e6876888-1550574921303@3c-app-webde-bs65>
	<CAMYG4GmNbNmWER_pgYeLFDkpV699GEb5S1FgVv-Avm3trOqZgg@mail.gmail.com>
	<trinity-1e233212-1706-40a0-8b4f-a04e258b9661-1550621535839@3c-app-webde-bs21>
	<CAMYG4Gm6CCZp=zE8MtZhtKruJ0O0O=nyT5nG0P=5hDerKDsCnQ@mail.gmail.com>
	<trinity-3a2ccd9b-25fa-4695-a866-6a07995cefae-1550626933255@3c-app-webde-bs38>
	<CAMYG4Gkr=jfY=tqqAukEBtmMMvtKuEwhLfBvAaw_j54Q41o5Kg@mail.gmail.com>
	<trinity-4ea6b41e-2343-4b4e-9dc3-e21eb1c83c54-1550707063864@3c-app-webde-bap44>
Message-ID: <CAMYG4Gk+o6EeDvDszbg=i8QwgvfMCYgsPuN54+FyBgDPyGLR+g@mail.gmail.com>

On Wed, Feb 20, 2019 at 6:57 PM Marius Buerkle <mbuerkle at web.de> wrote:

> ok, I think I understand now. I will give it a try and if there is some
> trouble comeback to you. thanks.
>

Cool.

   Matt


>
> marius
>
> On Tue, Feb 19, 2019 at 8:42 PM Marius Buerkle <mbuerkle at web.de> wrote:
>
>> ok, so it seems there is no straight forward way to transfer data between
>> PETSc matrices on different subcomms. Probably doing it by "hand"
>> extracting the matricies on the subcomms create a MPI_INTERCOMM transfering
>> the data to PETSC_COMM_WORLD and assembling them in a new PETSc matrix
>> would be possible, right?
>>
>
> That sounds too complicated. Why not just reverse
> MPICreateSubMatricesMPI()? Meaning make it collective on the whole big
> communicator, so that you can swap out all the subcommunicator for the
> aggregation call, just like we do in that function.
> Then its really just a matter of reversing the communication call.
>
>    Matt
>
>>
>> On Tue, Feb 19, 2019 at 7:12 PM Marius Buerkle <mbuerkle at web.de> wrote:
>>
>>> I see. This would work if the matrices are on different subcommumicators
>>> ? Is it possible to add this functionality ?
>>>
>>
>> Hmm, no. That is specialized to serial matrices. You need the inverse of
>> MatCreateSubMatricesMPI().
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> marius
>>>
>>>
>>> You basically need the inverse of MatCreateSubmatrices(). I do not think
>>> we have that right now, but it could probably be done without too much
>>> trouble by looking at that code.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>> On Tue, Feb 19, 2019 at 6:15 AM Marius Buerkle via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hi !
>>>>
>>>> Is there some way to combine MatCompositeMerge
>>>> with MatCreateRedundantMatrix? I basically want to create copies of a
>>>> matrix from PETSC_COMM_WORLD to subcommunicators, do some work on each
>>>> subcommunicator and than gather the results back to PETSC_COMM_WORLD,
>>>> namely  I want to sum the  the invidual matrices from the subcommunicatos
>>>> component wise and get the resulting matrix on PETSC_COMM_WORLD. Is this
>>>> somehow possible without going through all the hassle of using MPI
>>>> directly?
>>>>
>>>> marius
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/cea4d2f4/attachment.html>

From zinlin.zinlin at gmail.com  Wed Feb 20 22:16:35 2019
From: zinlin.zinlin at gmail.com (Zin Lin)
Date: Wed, 20 Feb 2019 23:16:35 -0500
Subject: [petsc-users] DM error? same code working on local but not on cray
 system
Message-ID: <CAGqL4bLGbPp0fitNV-GogEHsDAF2k0HQLD+ax8bksuC1_PhTBw@mail.gmail.com>

Hi
I have a finite difference code based on petsc-3.6.4 working perfectly fine
on a local server (using about ~40 cpus)

but the exact same code and the exact same example on a cray system
compiled with petsc-3.7.4 is giving me the following error:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Partition in y direction is too fine! 0 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
[0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin Wed
Feb 20 21:59:21 2019
[0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
--known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
--known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
--known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
--known-memcmp-ok=1 --known-mpi-c-double-complex=1
--known-mpi-long-double=0 --known-mpi-shared-libraries=0
--known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
--known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
--known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
--known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
--with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
--with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
--with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
--with-fortranlib-autodetect=0 --with-ranlib=ranlib
--with-scalar-type=complex --with-shared-ld=ar --with-etags=0
--with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
--with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
--with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
-lsci_cray_mp" --with-superlu_dist=1
--with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lsuperlu_dist-64" --with-parmetis=1
--with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lparmetis-64" --with-metis=1
--with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lmetis-64" --with-ptscotch=1
--with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
--with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/include --with-scalapack-lib="-L/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/lib -lsci_cray_mpi_mp -lsci_cray_mp"
--with-hdf5=1 --with-hdf5-include=/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/include --with-hdf5-lib="-L/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/lib -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2
-hpic" --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
--FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
--CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
[0]PETSC ERROR: #1 DMSetUp_DA_3D() line 298 in src/dm/impls/da/da3.c
[0]PETSC ERROR: #2 DMSetUp_DA() line 27 in src/dm/impls/da/dareg.c
[0]PETSC ERROR: #3 DMSetUp() line 744 in src/dm/interface/dm.c
[0]PETSC ERROR: #4 DMDACreate3d() line 1446 in src/dm/impls/da/da3.c
[0]PETSC ERROR: #5 setParDataGrid() line 176 in
/p/home/zinlin/Dev_3DOpt/library/initialize.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: key 1099516865361 is greater than largest key allowed 751
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
[0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin Wed
Feb 20 21:59:21 2019
[0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
--known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
--known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
--known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
--known-memcmp-ok=1 --known-mpi-c-double-complex=1
--known-mpi-long-double=0 --known-mpi-shared-libraries=0
--known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
--known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
--known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
--known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
--with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
--with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
--with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
--with-fortranlib-autodetect=0 --with-ranlib=ranlib
--with-scalar-type=complex --with-shared-ld=ar --with-etags=0
--with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
--with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
--with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
-lsci_cray_mp" --with-superlu_dist=1
--with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lsuperlu_dist-64" --with-parmetis=1
--with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lparmetis-64" --with-metis=1
--with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lmetis-64" --with-ptscotch=1
--with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
--with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/include --with-scalapack-lib="-L/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/lib -lsci_cray_mpi_mp -lsci_cray_mp"
--with-hdf5=1 --with-hdf5-include=/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/include --with-hdf5-lib="-L/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/lib -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2
-hpic" --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
--FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
--CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
[0]PETSC ERROR: #6 PetscTableFind() line 142 in
/b/cray-petsc/.cray-build/CRAY/86/x86_64/cray-petsc-64-complex/petsc-3.7.6/./include/petscctable.h
[0]PETSC ERROR: #7 MatSetUpMultiply_MPIAIJ() line 33 in
src/mat/impls/aij/mpi/mmaij.c
[0]PETSC ERROR: #8 MatAssemblyEnd_MPIAIJ() line 747 in
src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #9 MatAssemblyEnd() line 5194 in src/mat/interface/matrix.c
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
[0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin Wed
Feb 20 21:59:21 2019
[0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
--known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
--known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
--known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
--known-memcmp-ok=1 --known-mpi-c-double-complex=1
--known-mpi-long-double=0 --known-mpi-shared-libraries=0
--known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
--known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
--known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
--known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
--with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
--with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
--with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
--with-fortranlib-autodetect=0 --with-ranlib=ranlib
--with-scalar-type=complex --with-shared-ld=ar --with-etags=0
--with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
--with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
--with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
-lsci_cray_mp" --with-superlu_dist=1
--with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lsuperlu_dist-64" --with-parmetis=1
--with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lparmetis-64" --with-metis=1
--with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lmetis-64" --with-ptscotch=1
--with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
--with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
-lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
--with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/include --with-scalapack-lib="-L/opt/cray/libsci/
17.06.1.1/CRAY/8.6/x86_64/lib -lsci_cray_mpi_mp -lsci_cray_mp"
--with-hdf5=1 --with-hdf5-include=/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/include --with-hdf5-lib="-L/opt/cray/hdf5-parallel/
1.10.0.3/CRAY/8.6/lib -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2
-hpic" --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
--FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
--CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
[0]PETSC ERROR: #10 User provided function() line 0 in  unknown file
Rank 0 [Wed Feb 20 21:59:21 2019] [c13-0c0s8n3] application called
MPI_Abort(MPI_COMM_WORLD, 59) - process 0
_pmiu_daemon(SIGCHLD): [NID 02531] [c13-0c0s8n3] [Wed Feb 20 21:59:21 2019]
PE RANK 0 exit signal Aborted

now this is testing with just 1 cpu.
does it have something to do with DM having a problem because y dimension
is just one pixel (so a 2d problem though DMDAcreate3d was used) ?
but there was no problem with it before on the local server and petsc-3.6.4.
any help will be appreciated.
Thanks,
Zin

-- 
Zin Lin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/829c8304/attachment-0001.html>

From alia at wias-berlin.de  Thu Feb 21 09:46:53 2019
From: alia at wias-berlin.de (Najib Alia)
Date: Thu, 21 Feb 2019 16:46:53 +0100
Subject: [petsc-users] Using PETSc in Cray systems
Message-ID: <18a4fe1f-1b2e-4dc9-ba95-f0f4d00138be@wias-berlin.de>

Dear all,

we are trying to compile our Finite Element code on a Cray system and
have a problem with PETSc and available packages: "unable to find
scotch64", the variable PETSC_SINGLE_LIBRARY is set to NOTFOUND, and the
compiler tests fail.

What we have done:

1) We loaded the corresponding modules:

module load cray-tpsl

module load cray-petsc-64

2) and set the PETSC_DIR and ARCH variables as given in "module display
cray-petsc-64":

-------------------------------------------------------------------
/opt/cray/modulefiles/cray-petsc-64/3.7.4.0:

setenv?????????? PRGENV64 64
conflict???????? cray-petsc-complex
conflict???????? petsc-complex
conflict???????? cray-petsc
conflict???????? petsc
conflict???????? cray-hdf5
setenv?????????? CRAY_PETSC_DIR /opt/cray/petsc/3.7.4.0/real
setenv?????????? CRAY_PETSC_BASE_DIR /opt/cray/petsc/3.7.4.0/real
setenv?????????? CRAY_PETSC_VERSION 3.7.4.0
setenv?????????? CRAY_PETSC_PREFIX_DIR
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
setenv?????????? PETSC_DIR
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
setenv?????????? PETSC_ARCH
prepend-path???? PE_PRODUCT_LIST CRAY_PETSC
prepend-path???? CRAY_LD_LIBRARY_PATH
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge/lib
prepend-path???? MANPATH
/opt/cray/petsc/3.7.4.0/real/man:/opt/cray/man/csmlversion
setenv?????????? PE_PETSC_MODULE_NAME cray-petsc-64/3.7.4.0
setenv?????????? PE_PETSC_VOLATILE_PRGENV CRAY CRAY64 GNU GNU64 INTEL
INTEL64
...
module-whatis??? PETSc - Portable, Extensible Toolkit for Scientific
Computation
-------------------------------------------------------------------

3) the library scotch is correctly installed (verified using "module
display cray-tpsl").

Our program uses the following file to find PETSc:

https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake

Do you have suggestions ?

How can we check the current configuration of PETSc and set the
variables correctly? Is it normal that PETSC_ARCH is empty?

Since we do not have much experience with Cray systems, we would like to
avoid configure PETSc by our own.

Thanks ahead for your suggestions,

N. Alia


From t.appel17 at imperial.ac.uk  Thu Feb 21 10:09:44 2019
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Thu, 21 Feb 2019 16:09:44 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
Message-ID: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>

Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite 
differences, assembling an MPIAIJ matrix to solve linear systems or 
eigenvalue problems. I?ve been using vertex major, natural ordering for 
the parallelism with PetscSplitOwnership (yielding rectangular slices of 
the physical domain) and wanted to move to DMDA to have a more 
square-ish domain decomposition and minimize communication between 
processes.

However, my application is memory critical, and I have finely-tuned 
matrix preallocation routines for allocating memory ?optimally?. It 
seems the memory of a DMDA matrix is allocated along the value of the 
stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite 
difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not 
understand in the DM topology, what is that "stencil width" for then? I 
will not use ghost values for my FD-method, right?

I was then wondering if I could just create a MPIAIJ matrix, and with a 
PETSc routine get the global indices of the domain for each process: in 
other words, an equivalent of PetscSplitOwnership that gives me the DMDA 
unknown ordering. So I can feed and loop on that in my preallocation and 
assembly routines.

Thanks very much,

Thibaut
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/80caa28e/attachment.html>

From knepley at gmail.com  Thu Feb 21 11:44:05 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 21 Feb 2019 12:44:05 -0500
Subject: [petsc-users] Using PETSc in Cray systems
In-Reply-To: <18a4fe1f-1b2e-4dc9-ba95-f0f4d00138be@wias-berlin.de>
References: <18a4fe1f-1b2e-4dc9-ba95-f0f4d00138be@wias-berlin.de>
Message-ID: <CAMYG4GkMf9bc4tCkP4EYBODFhGU1V1kexJdrc5M8AgQLPqLaWg@mail.gmail.com>

On Thu, Feb 21, 2019 at 10:46 AM Najib Alia via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear all,
>
> we are trying to compile our Finite Element code on a Cray system and
> have a problem with PETSc and available packages: "unable to find
> scotch64", the variable PETSC_SINGLE_LIBRARY is set to NOTFOUND, and the
> compiler tests fail.
>

First, we should see if PETSc works correctly.

  mkdir tmp
  cd tmp
  <create simple makefile>
  ex5: ex5.o
    ${CLINKER} -o ex5 ex5.o ${PETSC_LIB}

  include ${PETSC_DIR}/lib/petsc/conf/variables
  include ${PETSC_DIR}/lib/petsc/conf/rules
  <Get source from
https://bitbucket.org/petsc/petsc/src/master/src/snes/examples/tutorials/ex5.c
>
  <Build and run>
  make ex5
  ./ex5 -snes_monitor

If that runs, it looks like the errors are coming from your own build
system, which I don't think we
would understand.

  Thanks,

    Matt

What we have done:
>
> 1) We loaded the corresponding modules:
>
> module load cray-tpsl
>
> module load cray-petsc-64
>
> 2) and set the PETSC_DIR and ARCH variables as given in "module display
> cray-petsc-64":
>
> -------------------------------------------------------------------
> /opt/cray/modulefiles/cray-petsc-64/3.7.4.0:
>
> setenv           PRGENV64 64
> conflict         cray-petsc-complex
> conflict         petsc-complex
> conflict         cray-petsc
> conflict         petsc
> conflict         cray-hdf5
> setenv           CRAY_PETSC_DIR /opt/cray/petsc/3.7.4.0/real
> setenv           CRAY_PETSC_BASE_DIR /opt/cray/petsc/3.7.4.0/real
> setenv           CRAY_PETSC_VERSION 3.7.4.0
> setenv           CRAY_PETSC_PREFIX_DIR
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
> setenv           PETSC_DIR
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
> setenv           PETSC_ARCH
> prepend-path     PE_PRODUCT_LIST CRAY_PETSC
> prepend-path     CRAY_LD_LIBRARY_PATH
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge/lib
> prepend-path     MANPATH
> /opt/cray/petsc/3.7.4.0/real/man:/opt/cray/man/csmlversion
> setenv           PE_PETSC_MODULE_NAME cray-petsc-64/3.7.4.0
> setenv           PE_PETSC_VOLATILE_PRGENV CRAY CRAY64 GNU GNU64 INTEL
> INTEL64
> ...
> module-whatis    PETSc - Portable, Extensible Toolkit for Scientific
> Computation
> -------------------------------------------------------------------
>
> 3) the library scotch is correctly installed (verified using "module
> display cray-tpsl").
>
> Our program uses the following file to find PETSc:
>
> https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake
>
> Do you have suggestions ?
>
> How can we check the current configuration of PETSc and set the
> variables correctly? Is it normal that PETSC_ARCH is empty?
>
> Since we do not have much experience with Cray systems, we would like to
> avoid configure PETSc by our own.
>
> Thanks ahead for your suggestions,
>
> N. Alia
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/74ffd0a1/attachment.html>

From knepley at gmail.com  Thu Feb 21 11:49:26 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 21 Feb 2019 12:49:26 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
Message-ID: <CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>

On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear PETSc developers/users,
>
> I?m solving linear PDEs on a regular grid with high-order finite
> differences, assembling an MPIAIJ matrix to solve linear systems or
> eigenvalue problems. I?ve been using vertex major, natural ordering for the
> parallelism with PetscSplitOwnership (yielding rectangular slices of the
> physical domain) and wanted to move to DMDA to have a more square-ish
> domain decomposition and minimize communication between processes.
>
> However, my application is memory critical, and I have finely-tuned matrix
> preallocation routines for allocating memory ?optimally?. It seems the
> memory of a DMDA matrix is allocated along the value of the stencil width
> of DMDACreate and the manual says about it
>
> ?These DMDA stencils have nothing directly to do with any finite
> difference stencils one might chose to use for a discretization?
>
> And despite reading the manual pages there must be something I do not
> understand in the DM topology, what is that "stencil width" for then? I
> will not use ghost values for my FD-method, right?
>

What this is saying is, "You might be using some stencil that is not STAR
or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems
you do, then you might be able to use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.


> I was then wondering if I could just create a MPIAIJ matrix, and with a
> PETSc routine get the global indices of the domain for each process: in
> other words, an equivalent of PetscSplitOwnership that gives me the DMDA
> unknown ordering. So I can feed and loop on that in my preallocation and
> assembly routines.
>

You can make an MPIAIJ matrix yourself of course. It should have the same
division of rows as the DMDA division of dofs. Also, MatSetValuesStencil()
will not work for a custom matrix.

  Thanks,

     Matt


> Thanks very much,
>
> Thibaut
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/66a00d3c/attachment.html>

From t.appel17 at imperial.ac.uk  Thu Feb 21 12:19:25 2019
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Thu, 21 Feb 2019 18:19:25 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
Message-ID: <174e7bb1-8a9e-277e-9d4c-16d3b149528f@imperial.ac.uk>

Hi Matthew,

Is your first part of your answer (using DMDASetBlockFills) valid only 
in the case I create a DMDA object?

Yes I think that is the kind of stencil I am using. I could know how the 
stencil looks like exactly, but I preallocate looping, for each process 
on all the elements of the stencil, grid node by grid node (which is not 
that costly, and "exact")

If I do NOT use a DMDA object and create my MPIAIJ matrix myself, how do 
I get the global row indices owned by the process (the "DMDA-like" ones 
you mentioned)? The problem is that MatGetOwnershipRange cannot be 
called if the matrix hasn't been preallocated, and I need the global 
indices to preallocate.

Thibaut


On 21/02/2019 17:49, Matthew Knepley wrote:
> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Dear PETSc developers/users,
>
>     I?m solving linear PDEs on a regular grid with high-order finite
>     differences, assembling an MPIAIJ matrix to solve linear systems
>     or eigenvalue problems. I?ve been using vertex major, natural
>     ordering for the parallelism with PetscSplitOwnership (yielding
>     rectangular slices of the physical domain) and wanted to move to
>     DMDA to have a more square-ish domain decomposition and minimize
>     communication between processes.
>
>     However, my application is memory critical, and I have
>     finely-tuned matrix preallocation routines for allocating memory
>     ?optimally?. It seems the memory of a DMDA matrix is allocated
>     along the value of the stencil width of DMDACreate and the manual
>     says about it
>
>     ?These DMDA stencils have nothing directly to do with any finite
>     difference stencils one might chose to use for a discretization?
>
>     And despite reading the manual pages there must be something I do
>     not understand in the DM topology, what is that "stencil width"
>     for then? I will not use ghost values for my FD-method, right?
>
>
> What this is saying is, "You might be using some stencil that is not 
> STAR or BOX, but we are preallocating according to one of those".
> If you really care about how much memory is preallocated, which it 
> seems you do, then you might be able to use
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>
> to tell use exactly how to preallocate.
>
>     I was then wondering if I could just create a MPIAIJ matrix, and
>     with a PETSc routine get the global indices of the domain for each
>     process: in other words, an equivalent of PetscSplitOwnership that
>     gives me the DMDA unknown ordering. So I can feed and loop on that
>     in my preallocation and assembly routines.
>
>
> You can make an MPIAIJ matrix yourself of course. It should have the 
> same division of rows as the DMDA division of dofs. Also, 
> MatSetValuesStencil() will not work for a custom matrix.
>
> ? Thanks,
>
> ? ? ?Matt
>
>     Thanks very much,
>
>     Thibaut
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/9d20ebf0/attachment.html>

From knepley at gmail.com  Thu Feb 21 12:40:36 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 21 Feb 2019 13:40:36 -0500
Subject: [petsc-users] DM error? same code working on local but not on
 cray system
In-Reply-To: <CAGqL4bLGbPp0fitNV-GogEHsDAF2k0HQLD+ax8bksuC1_PhTBw@mail.gmail.com>
References: <CAGqL4bLGbPp0fitNV-GogEHsDAF2k0HQLD+ax8bksuC1_PhTBw@mail.gmail.com>
Message-ID: <CAMYG4GkSykvioHj+=fgK8KQGCs+FSQEdCL5dS-ov0Ugfq43jWA@mail.gmail.com>

On Wed, Feb 20, 2019 at 11:19 PM Zin Lin via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi
> I have a finite difference code based on petsc-3.6.4 working perfectly
> fine on a local server (using about ~40 cpus)
>
> but the exact same code and the exact same example on a cray system
> compiled with petsc-3.7.4 is giving me the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Partition in y direction is too fine! 0 1
>

Okay, what this seem to be saying is that you have 1 point in z (which it
seems you do), but
zero processors in that direction. If you set P = 1, then this should go
away.

  Thanks,

    Matt


> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
> [0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin
> Wed Feb 20 21:59:21 2019
> [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
> --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
> --known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
> --known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
> --known-memcmp-ok=1 --known-mpi-c-double-complex=1
> --known-mpi-long-double=0 --known-mpi-shared-libraries=0
> --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
> --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
> --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
> --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
> --with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
> --with-fortranlib-autodetect=0 --with-ranlib=ranlib
> --with-scalar-type=complex --with-shared-ld=ar --with-etags=0
> --with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
> --with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
> --with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mp" --with-superlu_dist=1
> --with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lsuperlu_dist-64" --with-parmetis=1
> --with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lparmetis-64" --with-metis=1
> --with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lmetis-64" --with-ptscotch=1
> --with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
> --with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
> 17.06.1.1/CRAY/8.6/x86_64/include
> --with-scalapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mpi_mp -lsci_cray_mp" --with-hdf5=1
> --with-hdf5-include=/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/include
> --with-hdf5-lib="-L/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/lib
> -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2  -hpic"
> --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
> --FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
> --CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
> 3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 298 in src/dm/impls/da/da3.c
> [0]PETSC ERROR: #2 DMSetUp_DA() line 27 in src/dm/impls/da/dareg.c
> [0]PETSC ERROR: #3 DMSetUp() line 744 in src/dm/interface/dm.c
> [0]PETSC ERROR: #4 DMDACreate3d() line 1446 in src/dm/impls/da/da3.c
> [0]PETSC ERROR: #5 setParDataGrid() line 176 in
> /p/home/zinlin/Dev_3DOpt/library/initialize.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: key 1099516865361 is greater than largest key allowed 751
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
> [0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin
> Wed Feb 20 21:59:21 2019
> [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
> --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
> --known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
> --known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
> --known-memcmp-ok=1 --known-mpi-c-double-complex=1
> --known-mpi-long-double=0 --known-mpi-shared-libraries=0
> --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
> --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
> --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
> --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
> --with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
> --with-fortranlib-autodetect=0 --with-ranlib=ranlib
> --with-scalar-type=complex --with-shared-ld=ar --with-etags=0
> --with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
> --with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
> --with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mp" --with-superlu_dist=1
> --with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lsuperlu_dist-64" --with-parmetis=1
> --with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lparmetis-64" --with-metis=1
> --with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lmetis-64" --with-ptscotch=1
> --with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
> --with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
> 17.06.1.1/CRAY/8.6/x86_64/include
> --with-scalapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mpi_mp -lsci_cray_mp" --with-hdf5=1
> --with-hdf5-include=/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/include
> --with-hdf5-lib="-L/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/lib
> -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2  -hpic"
> --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
> --FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
> --CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
> 3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
> [0]PETSC ERROR: #6 PetscTableFind() line 142 in
> /b/cray-petsc/.cray-build/CRAY/86/x86_64/cray-petsc-64-complex/petsc-3.7.6/./include/petscctable.h
> [0]PETSC ERROR: #7 MatSetUpMultiply_MPIAIJ() line 33 in
> src/mat/impls/aij/mpi/mmaij.c
> [0]PETSC ERROR: #8 MatAssemblyEnd_MPIAIJ() line 747 in
> src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #9 MatAssemblyEnd() line 5194 in src/mat/interface/matrix.c
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
> [0]PETSC ERROR: ./farfieldopt_exec on a x86_64 named nid02531 by zinlin
> Wed Feb 20 21:59:21 2019
> [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1
> --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-sdot-returns-double=0
> --known-snrm2-returns-double=0 --known-level1-dcache-assoc=4
> --known-level1-dcache-linesize=64 --known-level1-dcache-size=16384
> --known-memcmp-ok=1 --known-mpi-c-double-complex=1
> --known-mpi-long-double=0 --known-mpi-shared-libraries=0
> --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1
> --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4
> --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2
> --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=1
> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0
> --with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0
> --with-fortranlib-autodetect=0 --with-ranlib=ranlib
> --with-scalar-type=complex --with-shared-ld=ar --with-etags=0
> --with-dependencies=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0
> --with-dependencies=0 --with-mpi-lib="[]" --with-mpi-include="[]"
> --with-blas-lapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mp" --with-superlu_dist=1
> --with-superlu_dist-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-superlu_dist-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lsuperlu_dist-64" --with-parmetis=1
> --with-parmetis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-parmetis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lparmetis-64" --with-metis=1
> --with-metis-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-metis-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lmetis-64" --with-ptscotch=1
> --with-ptscotch-include=/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/include
> --with-ptscotch-lib="-L/opt/cray/tpsl/17.06.1/CRAY64/8.6/x86_64/lib
> -lptscotch-64 -lscotch-64 -lptscotcherr-64 -lscotcherr-64"
> --with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/
> 17.06.1.1/CRAY/8.6/x86_64/include
> --with-scalapack-lib="-L/opt/cray/libsci/17.06.1.1/CRAY/8.6/x86_64/lib
> -lsci_cray_mpi_mp -lsci_cray_mp" --with-hdf5=1
> --with-hdf5-include=/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/include
> --with-hdf5-lib="-L/opt/cray/hdf5-parallel/1.10.0.3/CRAY/8.6/lib
> -lhdf5_parallel -lz -ldl" --CFLAGS="-hnopattern -O2  -hpic"
> --CPPFLAGS="-hnomessage=11709 " --CXXFLAGS="-hnopattern -O2   -hpic"
> --FFLAGS="-hnopattern -O2  -F -em -hnocaf  -hpic" --LIBS=
> --CXX_LINKER_FLAGS= --PETSC_ARCH=x86_64 --prefix=/opt/cray/pe/petsc/
> 3.7.6.0/complex/CRAY64/8.6/x86_64 --with-64-bit-indices
> [0]PETSC ERROR: #10 User provided function() line 0 in  unknown file
> Rank 0 [Wed Feb 20 21:59:21 2019] [c13-0c0s8n3] application called
> MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> _pmiu_daemon(SIGCHLD): [NID 02531] [c13-0c0s8n3] [Wed Feb 20 21:59:21
> 2019] PE RANK 0 exit signal Aborted
>
> now this is testing with just 1 cpu.
> does it have something to do with DM having a problem because y dimension
> is just one pixel (so a 2d problem though DMDAcreate3d was used) ?
> but there was no problem with it before on the local server and
> petsc-3.6.4.
> any help will be appreciated.
> Thanks,
> Zin
>
> --
> Zin Lin
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/6438edd3/attachment-0001.html>

From qiming2 at illinois.edu  Thu Feb 21 14:23:30 2019
From: qiming2 at illinois.edu (Zhu, Qiming)
Date: Thu, 21 Feb 2019 20:23:30 +0000
Subject: [petsc-users] Question with filedsplit in PETSc
Message-ID: <CY4PR11MB00569BEE94241D9905B1C8389C7E0@CY4PR11MB0056.namprd11.prod.outlook.com>


Dear all,


Sorry to disturb you. I am a user of Petsc. I am trying to use Fieldsplit in Petsc to do preconditioning for Navier-Stokes problem. I have some problems when I trying to use Fieldsplit function. I am now defining the nest matrix first, then I get the IS from nested matrix. But I find that my code just work for one core. When I change to parallel case, I could only get zero solution. I wonder is there any special requirements for IS definition in Fieldsplit? I include one code here. If you have any idea, hope you reply soon. Thank you for your help. Thank you very much.


Yours sincerely,

Qiming Zhu,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/20e30bc8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 550 bytes
Desc: makefile
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/20e30bc8/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5.c
Type: text/x-csrc
Size: 4719 bytes
Desc: ex5.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190221/20e30bc8/attachment.bin>

From bsmith at mcs.anl.gov  Thu Feb 21 16:51:56 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 21 Feb 2019 22:51:56 +0000
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
Message-ID: <10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>


  Randy,

    Could you please cut and paste the entire error message you get. It worked for me. 

    I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).

  Satish,

      I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?

   Thanks

   Barry


> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
> We have installed PETSc following the instructions here:
> https://www.mcs.anl.gov/petsc/features/gpus.html
> 
> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
> 
> Is it possible to run PETSc on GPUS in complex mode?d
> 
> 
> Thanks, Randy M.


From rlmackie862 at gmail.com  Thu Feb 21 18:20:12 2019
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 21 Feb 2019 16:20:12 -0800
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
Message-ID: <2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>

Hi Barry and Satish,

Yes, sorry, I meant -dm_mat_type_aijcusparse?..

Here is an attempt to run ex39 under complex:

0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
[0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
[0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0


I used these options:

#!/bin/bash

export PETSC_ARCH=linux-gfortran-complex-debug

${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
-ksp_type bcgs \
-ksp_rtol 1.e-6 \
-pc_type jacobi \
-ksp_monitor_true_residual \
-ksp_converged_reason \
-mat_type aijcusparse \
-vec_type cuda \
-n1 32 \
-n2 32 \
-n3 32 \


My next step was going to try valgrind and see if that turned something up.

Thanks, Randy


> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Randy,
> 
>    Could you please cut and paste the entire error message you get. It worked for me. 
> 
>    I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
> 
>  Satish,
> 
>      I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
> 
>   Thanks
> 
>   Barry
> 
> 
>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>> 
>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>> We have installed PETSc following the instructions here:
>> https://www.mcs.anl.gov/petsc/features/gpus.html
>> 
>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>> 
>> Is it possible to run PETSc on GPUS in complex mode?d
>> 
>> 
>> Thanks, Randy M.
> 


From bsmith at mcs.anl.gov  Thu Feb 21 23:53:06 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 22 Feb 2019 05:53:06 +0000
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
	<2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
Message-ID: <61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>


   Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and it runs though the -n1 n2 n3 options aren't used.

   Barry


> On Feb 21, 2019, at 6:20 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> Hi Barry and Satish,
> 
> Yes, sorry, I meant -dm_mat_type_aijcusparse?..
> 
> Here is an attempt to run ex39 under complex:
> 
> 0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
> [0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> 
> I used these options:
> 
> #!/bin/bash
> 
> export PETSC_ARCH=linux-gfortran-complex-debug
> 
> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
> -ksp_type bcgs \
> -ksp_rtol 1.e-6 \
> -pc_type jacobi \
> -ksp_monitor_true_residual \
> -ksp_converged_reason \
> -mat_type aijcusparse \
> -vec_type cuda \
> -n1 32 \
> -n2 32 \
> -n3 32 \
> 
> 
> My next step was going to try valgrind and see if that turned something up.
> 
> Thanks, Randy
> 
> 
>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>> Randy,
>> 
>>   Could you please cut and paste the entire error message you get. It worked for me. 
>> 
>>   I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
>> 
>> Satish,
>> 
>>     I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
>> 
>>  Thanks
>> 
>>  Barry
>> 
>> 
>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> 
>>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>>> We have installed PETSc following the instructions here:
>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>> 
>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>>> 
>>> Is it possible to run PETSc on GPUS in complex mode?d
>>> 
>>> 
>>> Thanks, Randy M.
>> 
> 


From t.appel17 at imperial.ac.uk  Fri Feb 22 07:53:07 2019
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Fri, 22 Feb 2019 13:53:07 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
Message-ID: <4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>

I reckon that what corresponds best to my situation is to create my own 
MPIAIJ matrix, recover the DMDA local to global mapping, preallocate 
myself and fill my values with MatSetValues.

However, what is the correct way to call 
ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something 
missing in the documentation as my code just won't work.

Some examples include a PetscOffset argument which is not present in the 
documentation and there is no manual page for 
ISLocalToGlobalMappingGetIndicesF90: the only information I could find 
is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to 
allocate the index array yourself? I tried declaring it as a pointer 
too. I keep getting an error

[0]PETSC ERROR: #1 User provided function() line 0 in User file
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0

Minimal example:

PROGRAM test_dmda

#include <petsc/finclude/petscdm.h>
#include <petsc/finclude/petscdmda.h>

 ? USE PetscDMDA

 ? IMPLICIT NONE

 ? PetscErrorCode :: ierr
 ? PetscInt?????? :: irank, icx, icy, lx, ly, n_ltog
 ? DM???????????? :: da
 ? ISLocalToGlobalMapping :: ltog

 ? PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
 ? !PetscInt, DIMENSION(:), POINTER :: id_ltog

 ? PetscInt, PARAMETER :: nx=24, ny=9, neq=4
 ? CHARACTER(LEN=100) :: err_msg

 ? CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)

 ? CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)

 ? CALL 
DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), 
&
PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
 ? CHKERRA(ierr)

 ? CALL DMSetUp(da,ierr)
 ? CHKERRA(ierr)

 ? CALL 
DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
 ? CHKERRA(ierr)

 ? WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left 
corner is icx=', icx, &
 ????????????????????????? 'and icy=', icy, ' - the width is lx=', lx, ' 
and ly=', ly

 ? CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
 ? CHKERRA(ierr)

 ? CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
 ? CHKERRA(ierr)

 ? ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)

 ? CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
 ? CHKERRA(ierr)

 ? CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
 ? CHKERRA(ierr)

 ? DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)

 ? CALL DMDestroy(da,ierr)
 ? CHKERRA(ierr)

 ? CALL PetscFinalize(ierr)

END PROGRAM test_dmda


On 21/02/2019 17:49, Matthew Knepley wrote:
> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Dear PETSc developers/users,
>
>     I?m solving linear PDEs on a regular grid with high-order finite
>     differences, assembling an MPIAIJ matrix to solve linear systems
>     or eigenvalue problems. I?ve been using vertex major, natural
>     ordering for the parallelism with PetscSplitOwnership (yielding
>     rectangular slices of the physical domain) and wanted to move to
>     DMDA to have a more square-ish domain decomposition and minimize
>     communication between processes.
>
>     However, my application is memory critical, and I have
>     finely-tuned matrix preallocation routines for allocating memory
>     ?optimally?. It seems the memory of a DMDA matrix is allocated
>     along the value of the stencil width of DMDACreate and the manual
>     says about it
>
>     ?These DMDA stencils have nothing directly to do with any finite
>     difference stencils one might chose to use for a discretization?
>
>     And despite reading the manual pages there must be something I do
>     not understand in the DM topology, what is that "stencil width"
>     for then? I will not use ghost values for my FD-method, right?
>
>
> What this is saying is, "You might be using some stencil that is not 
> STAR or BOX, but we are preallocating according to one of those".
> If you really care about how much memory is preallocated, which it 
> seems you do, then you might be able to use
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>
> to tell use exactly how to preallocate.
>
>     I was then wondering if I could just create a MPIAIJ matrix, and
>     with a PETSc routine get the global indices of the domain for each
>     process: in other words, an equivalent of PetscSplitOwnership that
>     gives me the DMDA unknown ordering. So I can feed and loop on that
>     in my preallocation and assembly routines.
>
>
> You can make an MPIAIJ matrix yourself of course. It should have the 
> same division of rows as the DMDA division of dofs. Also, 
> MatSetValuesStencil() will not work for a custom matrix.
>
> ? Thanks,
>
> ? ? ?Matt
>
>     Thanks very much,
>
>     Thibaut
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190222/3b60dc7d/attachment.html>

From knepley at gmail.com  Fri Feb 22 08:23:57 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 22 Feb 2019 09:23:57 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <174e7bb1-8a9e-277e-9d4c-16d3b149528f@imperial.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<174e7bb1-8a9e-277e-9d4c-16d3b149528f@imperial.ac.uk>
Message-ID: <CAMYG4Gnp6bn9so6ZU8oegp_X8ai6R2psB1kvB8-QwK1J3fR3qA@mail.gmail.com>

On Thu, Feb 21, 2019 at 1:19 PM Thibaut Appel <t.appel17 at imperial.ac.uk>
wrote:

> Hi Matthew,
>
> Is your first part of your answer (using DMDASetBlockFills) valid only in
> the case I create a DMDA object?
>
> Yes I think that is the kind of stencil I am using. I could know how the
> stencil looks like exactly, but I preallocate looping, for each process on
> all the elements of the stencil, grid node by grid node (which is not that
> costly, and "exact")
>
> If I do NOT use a DMDA object and create my MPIAIJ matrix myself, how do I
> get the global row indices owned by the process (the "DMDA-like" ones you
> mentioned)?
>
I think you want


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetLocalInfo.html

This tells you exactly what patch is owned by the process, and also how big
the overlap is. Its not the same as G2L, but do you need
the whole G2L, or just the sizes?

  Thanks,

    Matt


> The problem is that MatGetOwnershipRange cannot be called if the matrix
> hasn't been preallocated, and I need the global indices to preallocate.
>
> Thibaut
>
>
> On 21/02/2019 17:49, Matthew Knepley wrote:
>
> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear PETSc developers/users,
>>
>> I?m solving linear PDEs on a regular grid with high-order finite
>> differences, assembling an MPIAIJ matrix to solve linear systems or
>> eigenvalue problems. I?ve been using vertex major, natural ordering for the
>> parallelism with PetscSplitOwnership (yielding rectangular slices of the
>> physical domain) and wanted to move to DMDA to have a more square-ish
>> domain decomposition and minimize communication between processes.
>>
>> However, my application is memory critical, and I have finely-tuned
>> matrix preallocation routines for allocating memory ?optimally?. It seems
>> the memory of a DMDA matrix is allocated along the value of the stencil
>> width of DMDACreate and the manual says about it
>>
>> ?These DMDA stencils have nothing directly to do with any finite
>> difference stencils one might chose to use for a discretization?
>>
>> And despite reading the manual pages there must be something I do not
>> understand in the DM topology, what is that "stencil width" for then? I
>> will not use ghost values for my FD-method, right?
>>
>
> What this is saying is, "You might be using some stencil that is not STAR
> or BOX, but we are preallocating according to one of those".
> If you really care about how much memory is preallocated, which it seems
> you do, then you might be able to use
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>
> to tell use exactly how to preallocate.
>
>
>> I was then wondering if I could just create a MPIAIJ matrix, and with a
>> PETSc routine get the global indices of the domain for each process: in
>> other words, an equivalent of PetscSplitOwnership that gives me the DMDA
>> unknown ordering. So I can feed and loop on that in my preallocation and
>> assembly routines.
>>
>
> You can make an MPIAIJ matrix yourself of course. It should have the same
> division of rows as the DMDA division of dofs. Also, MatSetValuesStencil()
> will not work for a custom matrix.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks very much,
>>
>> Thibaut
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190222/423f4659/attachment.html>

From t.appel17 at imperial.ac.uk  Fri Feb 22 09:05:48 2019
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Fri, 22 Feb 2019 15:05:48 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4Gnp6bn9so6ZU8oegp_X8ai6R2psB1kvB8-QwK1J3fR3qA@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<174e7bb1-8a9e-277e-9d4c-16d3b149528f@imperial.ac.uk>,
	<CAMYG4Gnp6bn9so6ZU8oegp_X8ai6R2psB1kvB8-QwK1J3fR3qA@mail.gmail.com>
Message-ID: <6D9410AB-B334-4991-87F5-02E9E3DAD7E0@imperial.ac.uk>

Hi Matthew,

I think I need the whole G2L not just the sizes.
Build my own matrix, create a ?fake? DMDA, extract the G2L mapping and apply it to my preallocation/assembly routines (where I would basically replace my natural ordering with the DMDA ordering from the G2L mapping)

(cf the mail I sent to the user list approx. 1 hour ago)

Thibaut

On 22 Feb 2019, at 14:24, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Thu, Feb 21, 2019 at 1:19 PM Thibaut Appel <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:

Hi Matthew,

Is your first part of your answer (using DMDASetBlockFills) valid only in the case I create a DMDA object?

Yes I think that is the kind of stencil I am using. I could know how the stencil looks like exactly, but I preallocate looping, for each process on all the elements of the stencil, grid node by grid node (which is not that costly, and "exact")

If I do NOT use a DMDA object and create my MPIAIJ matrix myself, how do I get the global row indices owned by the process (the "DMDA-like" ones you mentioned)?

I think you want

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetLocalInfo.html

This tells you exactly what patch is owned by the process, and also how big the overlap is. Its not the same as G2L, but do you need
the whole G2L, or just the sizes?

  Thanks,

    Matt


The problem is that MatGetOwnershipRange cannot be called if the matrix hasn't been preallocated, and I need the global indices to preallocate.

Thibaut


On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I?ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.

However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory ?optimally?. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?

What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop on that in my preallocation and assembly routines.

You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.

  Thanks,

     Matt

Thanks very much,

Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190222/14a05e70/attachment-0001.html>

From knepley at gmail.com  Fri Feb 22 09:13:49 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 22 Feb 2019 10:13:49 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
Message-ID: <CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>

On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> I reckon that what corresponds best to my situation is to create my own
> MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself
> and fill my values with MatSetValues.
>
> However, what is the correct way to call ISLocalToGlobalMappingGetIndices
> in Fortran? It seems there's something missing in the documentation as my
> code just won't work.
>
> Some examples include a PetscOffset argument which is not present in the
> documentation and there is no manual page for
> ISLocalToGlobalMappingGetIndicesF90: the only information I could find is
> on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to allocate
> the index array yourself? I tried declaring it as a pointer too. I keep
> getting an error
>
1) There was a bug in that file, but I don't think it should have affected
you. I will put in a PR

2)  You are supposed to pass in a pointer, unallocated. We manage all the
allocation, which is why you call Restore. What happens when you do that?

  Thanks,

     Matt

> [0]PETSC ERROR: #1 User provided function() line 0 in User file
> application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0
>
> Minimal example:
>
> PROGRAM test_dmda
>
> #include <petsc/finclude/petscdm.h>
> #include <petsc/finclude/petscdmda.h>
>
>   USE PetscDMDA
>
>   IMPLICIT NONE
>
>   PetscErrorCode :: ierr
>   PetscInt       :: irank, icx, icy, lx, ly, n_ltog
>   DM             :: da
>   ISLocalToGlobalMapping :: ltog
>
>   PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
>   !PetscInt, DIMENSION(:), POINTER :: id_ltog
>
>   PetscInt, PARAMETER :: nx=24, ny=9, neq=4
>   CHARACTER(LEN=100) :: err_msg
>
>   CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>
>   CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)
>
>   CALL
> DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),
> &
>
> PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>   CHKERRA(ierr)
>
>   CALL DMSetUp(da,ierr)
>   CHKERRA(ierr)
>
>   CALL
> DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>   CHKERRA(ierr)
>
>   WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left
> corner is icx=', icx, &
>                           'and icy=', icy, ' - the width is lx=', lx, '
> and ly=', ly
>
>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
>   CHKERRA(ierr)
>
>   ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)
>
>   CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)
>
>   CALL DMDestroy(da,ierr)
>   CHKERRA(ierr)
>
>   CALL PetscFinalize(ierr)
>
> END PROGRAM test_dmda
>
>
> On 21/02/2019 17:49, Matthew Knepley wrote:
>
> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear PETSc developers/users,
>>
>> I?m solving linear PDEs on a regular grid with high-order finite
>> differences, assembling an MPIAIJ matrix to solve linear systems or
>> eigenvalue problems. I?ve been using vertex major, natural ordering for the
>> parallelism with PetscSplitOwnership (yielding rectangular slices of the
>> physical domain) and wanted to move to DMDA to have a more square-ish
>> domain decomposition and minimize communication between processes.
>>
>> However, my application is memory critical, and I have finely-tuned
>> matrix preallocation routines for allocating memory ?optimally?. It seems
>> the memory of a DMDA matrix is allocated along the value of the stencil
>> width of DMDACreate and the manual says about it
>>
>> ?These DMDA stencils have nothing directly to do with any finite
>> difference stencils one might chose to use for a discretization?
>>
>> And despite reading the manual pages there must be something I do not
>> understand in the DM topology, what is that "stencil width" for then? I
>> will not use ghost values for my FD-method, right?
>>
>
> What this is saying is, "You might be using some stencil that is not STAR
> or BOX, but we are preallocating according to one of those".
> If you really care about how much memory is preallocated, which it seems
> you do, then you might be able to use
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>
> to tell use exactly how to preallocate.
>
>
>> I was then wondering if I could just create a MPIAIJ matrix, and with a
>> PETSc routine get the global indices of the domain for each process: in
>> other words, an equivalent of PetscSplitOwnership that gives me the DMDA
>> unknown ordering. So I can feed and loop on that in my preallocation and
>> assembly routines.
>>
>
> You can make an MPIAIJ matrix yourself of course. It should have the same
> division of rows as the DMDA division of dofs. Also, MatSetValuesStencil()
> will not work for a custom matrix.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks very much,
>>
>> Thibaut
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190222/73218c20/attachment.html>

From rlmackie862 at gmail.com  Fri Feb 22 16:48:28 2019
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Fri, 22 Feb 2019 14:48:28 -0800
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
	<2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
	<61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>
Message-ID: <86A03191-5D39-47B7-8448-23CE677BD3A3@gmail.com>

Below is more systematic information about my attempts to run complex on a GPU. Thus far I have been unsuccessful, and any pointers/suggestions are appreciated. Next week I will try another machine with a different GPU.

In the meantime, here is what i get:


We have a machine on which we have 1 GPU:

GPU 0: GeForce GTX 1050 Ti 

We have installed the latest version of Cuda:

Cuda compilation tools, release 10.0, V10.0.130


I have compiled a gfortran version of petsc in complex:

./configure \
  --with-clean=1 \
  --with-scalar-type=complex \
  --with-debugging=1 \
  --with-fortran=1 \
  --with-cuda=1 \
  --with-cudac=/usr/local/cuda-10.0/bin/nvcc \
  --download-mpich=./mpich-3.3b1.tar.gz \
  --download-fblaslapack=fblaslapack-3.4.2.tar.gz \


To test I compiled ksp/ksp/examples/tests/ex32.c with a real version of petsc.
Run on the CPU with the following options I get the following output:

mpirun -np 1 ./ex32 \
  -dm_mat_type sbaij \
  -pc_type jacobi \
  -ksp_monitor_short

# ./cmd_ex32
  0 KSP Residual norm 0.076964 
  1 KSP Residual norm 0.0461195 
  2 KSP Residual norm 0.0281244 
  3 KSP Residual norm 0.0182566 
  4 KSP Residual norm 0.00902356 
  5 KSP Residual norm 0.00360423 
  6 KSP Residual norm 0.0010161 
  7 KSP Residual norm 0.000379953 
  8 KSP Residual norm 0.000168001 
  9 KSP Residual norm 0.000100787 
 10 KSP Residual norm 6.30212e-05 
 11 KSP Residual norm 3.74954e-05 
 12 KSP Residual norm 1.30554e-05 
 13 KSP Residual norm 3.7475e-06 
 14 KSP Residual norm 1.7272e-06 
 15 KSP Residual norm 6.5817e-07 


When I run on the GPU, I get the following:

${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex32_real \
  -dm_mat_type aijcusparse \
  -dm_vec_type cuda \
  -pc_type jacobi \
  -ksp_monitor_short \


# ./cmd_ex32_cuda 
  0 KSP Residual norm 0.076964 
  1 KSP Residual norm 0.0456494 
  2 KSP Residual norm 0.0277184 
  3 KSP Residual norm 0.0156403 
  4 KSP Residual norm 0.0077786 
  5 KSP Residual norm 0.00211375 
  6 KSP Residual norm 0.000615725 
  7 KSP Residual norm 0.000169699 
  8 KSP Residual norm 6.17985e-05 
  9 KSP Residual norm 1.57243e-05 
 10 KSP Residual norm 8.98356e-07 
 11 KSP Residual norm 1.51097e-07 

When run with the complex debug version I get this:

# ./cmd_ex32_cmplx 
  0 KSP Residual norm < 1.e-11

(obviously this is not correct, even if it does not bomb out).

Running this example under valgrind gives tons of unhelpful messages like this:

0 KSP Residual norm < 1.e-11
==3767== 4 bytes in 1 blocks are possibly lost in loss record 9 of 1,618
==3767==    at 0x4C29BC3: malloc (vg_replace_malloc.c:299)
==3767==    by 0x1A6EE927: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A7A0BE4: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A7ACD4F: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A7AD94A: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A81F045: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A81F301: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A81FAD7: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A82033D: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A69E8EB: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A6A092E: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A5D530B: ??? (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x1A70C5AA: cuDevicePrimaryCtxRetain (in /usr/lib64/libcuda.so.410.48)
==3767==    by 0x12376D2F: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x12376E9B: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x1237787E: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x123782E7: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x1236B43D: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x1235ADE7: ??? (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767==    by 0x1238C23B: cudaMalloc (in /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130)
==3767== 



When I try ex39.c in the same directory (ksp/ksp/examples/tests), under valgrind, I get this:

# ./cmd_ex39
==3684== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x7ff with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x37 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Warning: noted but unhandled ioctl 0x42 with no size/direction hints.
==3684==    This could cause spurious value errors to appear.
==3684==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3684== Invalid read of size 8
==3684==    at 0x50F7C6C: VecCUDAGetArrayRead (veccuda2.cu:1284)
==3684==    by 0x50EB8C3: VecAYPX_SeqCUDA (veccuda2.cu:188)
==3684==    by 0x52F741C: VecAYPX (rvector.c:750)
==3684==    by 0x6701CF5: KSPBuildResidualDefault (iterativ.c:891)
==3684==    by 0x66EA04E: KSPBuildResidual (itfunc.c:2142)
==3684==    by 0x66FB812: KSPMonitorTrueResidualNorm (iterativ.c:259)
==3684==    by 0x66E7272: KSPMonitor (itfunc.c:1716)
==3684==    by 0x6673AB4: KSPSolve_BCGS (bcgs.c:64)
==3684==    by 0x66DCA48: KSPSolve (itfunc.c:780)
==3684==    by 0x402601: main (ex39.c:111)
==3684==  Address 0xa0 is not stack'd, malloc'd or (recently) free'd
==3684== 
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
[0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
[0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Fri Feb 22 14:43:35 2019
[0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0


I also tried another example, snes/examples/tutorials/ex19.c, which gives similar messages:

# ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
  0 SNES Function norm 0.239155 
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAXPY_SeqCUDA line 215 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: [0] VecAXPY line 597 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: [0] MatFDColoringApply_AIJ line 176 /home/everderio/DEV/petsc-3.10.3/src/mat/impls/aij/mpi/fdmpiaij.c
[0]PETSC ERROR: [0] MatFDColoringApply line 614 /home/everderio/DEV/petsc-3.10.3/src/mat/matfd/fdmatrix.c
[0]PETSC ERROR: [0] SNESComputeJacobian_DMDA line 153 /home/everderio/DEV/petsc-3.10.3/src/snes/utils/dmdasnes.c
[0]PETSC ERROR: [0] SNES user Jacobian function line 2555 /home/everderio/DEV/petsc-3.10.3/src/snes/interface/snes.c
[0]PETSC ERROR: [0] SNESComputeJacobian line 2514 /home/everderio/DEV/petsc-3.10.3/src/snes/interface/snes.c
[0]PETSC ERROR: [0] SNESSolve_NEWTONLS line 144 /home/everderio/DEV/petsc-3.10.3/src/snes/impls/ls/ls.c
[0]PETSC ERROR: [0] SNESSolve line 4282 /home/everderio/DEV/petsc-3.10.3/src/snes/interface/snes.c
[0]PETSC ERROR: [0] main line 108 /home/everderio/DEV/petsc-3.10.3/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
[0]PETSC ERROR: ./ex19 on a linux-gfortran-complex-debug named GPU by root Fri Feb 22 17:24:34 2019
[0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_61 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
:
system msg for write_line failure : Bad file descriptor


Thanks for any help or suggestions about getting complex to work with GPUS.

Randy M.


> On Feb 21, 2019, at 9:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>   Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and it runs though the -n1 n2 n3 options aren't used.
> 
>   Barry
> 
> 
>> On Feb 21, 2019, at 6:20 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> Hi Barry and Satish,
>> 
>> Yes, sorry, I meant -dm_mat_type_aijcusparse?..
>> 
>> Here is an attempt to run ex39 under complex:
>> 
>> 0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>> [0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
>> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>> [0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
>> [0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
>> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
>> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>> 
>> 
>> I used these options:
>> 
>> #!/bin/bash
>> 
>> export PETSC_ARCH=linux-gfortran-complex-debug
>> 
>> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
>> -ksp_type bcgs \
>> -ksp_rtol 1.e-6 \
>> -pc_type jacobi \
>> -ksp_monitor_true_residual \
>> -ksp_converged_reason \
>> -mat_type aijcusparse \
>> -vec_type cuda \
>> -n1 32 \
>> -n2 32 \
>> -n3 32 \
>> 
>> 
>> My next step was going to try valgrind and see if that turned something up.
>> 
>> Thanks, Randy
>> 
>> 
>>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Randy,
>>> 
>>>  Could you please cut and paste the entire error message you get. It worked for me. 
>>> 
>>>  I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
>>> 
>>> Satish,
>>> 
>>>    I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
>>> 
>>> Thanks
>>> 
>>> Barry
>>> 
>>> 
>>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>> 
>>>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>>>> We have installed PETSc following the instructions here:
>>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>>> 
>>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
>>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>>>> 
>>>> Is it possible to run PETSc on GPUS in complex mode?d
>>>> 
>>>> 
>>>> Thanks, Randy M.
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Fri Feb 22 22:53:12 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 23 Feb 2019 04:53:12 +0000
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <8ED67D2D-451C-42CA-84CE-27E6FEE7E0A1@gmail.com>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
	<2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
	<61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>
	<8ED67D2D-451C-42CA-84CE-27E6FEE7E0A1@gmail.com>
Message-ID: <20733FC8-2437-4AC1-89D5-8CCB2E18397A@mcs.anl.gov>


  I am getting some crashes on our machine frog with complex cuda on those examples, though maybe not at the exact same line numbers.

  We need more help from our CUDA experts

   Barry


> On Feb 22, 2019, at 8:33 AM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> 
> Sorry, I?ve tried both ex32 and ex39 in the src/ksp/ksp/examples/tests directory, both would give the same error.
> I?ll try with valgrind or try another computer with a different GPU.
> 
> Thanks for confirming that complex *should* work on GPUs.
> 
> Randy M.
> 
> 
> 
>> On Feb 21, 2019, at 9:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and it runs though the -n1 n2 n3 options aren't used.
>> 
>>  Barry
>> 
>> 
>>> On Feb 21, 2019, at 6:20 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>> 
>>> Hi Barry and Satish,
>>> 
>>> Yes, sorry, I meant -dm_mat_type_aijcusparse?..
>>> 
>>> Here is an attempt to run ex39 under complex:
>>> 
>>> 0]PETSC ERROR: ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>> [0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>> [0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
>>> [0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
>>> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
>>> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>> 
>>> 
>>> I used these options:
>>> 
>>> #!/bin/bash
>>> 
>>> export PETSC_ARCH=linux-gfortran-complex-debug
>>> 
>>> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
>>> -ksp_type bcgs \
>>> -ksp_rtol 1.e-6 \
>>> -pc_type jacobi \
>>> -ksp_monitor_true_residual \
>>> -ksp_converged_reason \
>>> -mat_type aijcusparse \
>>> -vec_type cuda \
>>> -n1 32 \
>>> -n2 32 \
>>> -n3 32 \
>>> 
>>> 
>>> My next step was going to try valgrind and see if that turned something up.
>>> 
>>> Thanks, Randy
>>> 
>>> 
>>>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>> 
>>>> 
>>>> Randy,
>>>> 
>>>> Could you please cut and paste the entire error message you get. It worked for me. 
>>>> 
>>>> I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
>>>> 
>>>> Satish,
>>>> 
>>>>   I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
>>>> 
>>>> Thanks
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> 
>>>>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>>>>> We have installed PETSc following the instructions here:
>>>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>>>> 
>>>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
>>>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>>>>> 
>>>>> Is it possible to run PETSc on GPUS in complex mode?d
>>>>> 
>>>>> 
>>>>> Thanks, Randy M.
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Fri Feb 22 23:00:08 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 23 Feb 2019 05:00:08 +0000
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <20733FC8-2437-4AC1-89D5-8CCB2E18397A@mcs.anl.gov>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
	<2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
	<61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>
	<8ED67D2D-451C-42CA-84CE-27E6FEE7E0A1@gmail.com>
	<20733FC8-2437-4AC1-89D5-8CCB2E18397A@mcs.anl.gov>
Message-ID: <4AF22B0C-DC48-452E-BD6C-934B9E62B5C4@mcs.anl.gov>


   I get this in the debugger (can't make heads or tails out of it, does thrust handle complex in some way we don't understand so doesn't work with our source code?) 

Thread 1 "ex32" received signal SIGSEGV, Segmentation fault.
0x00007f1a3913abd0 in thrust::complex<double>::real (this=0x0) at /usr/local/cuda/include/thrust/complex.h:324
324	  T real() const { return data.x; }
(gdb) b
Breakpoint 1 at 0x7f1a3913abd0: file /usr/local/cuda/include/thrust/complex.h, line 324.
(gdb) bt
#0  0x00007f1a3913abd0 in thrust::complex<double>::real (this=0x0)
    at /usr/local/cuda/include/thrust/complex.h:324
#1  0x00007f1a3913ae48 in thrust::operator==<double, double> (x=..., y=...)
    at /usr/local/cuda/include/thrust/detail/complex/complex.inl:313
#2  0x00007f1a3912f8ff in VecSet_SeqCUDA (xin=0x215c5e0, 
    alpha=<error reading variable: Cannot access memory at address 0x0>)
    at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda2.cu:775
#3  0x00007f1a39336ee7 in VecSet (x=0x215c5e0, alpha=0 + 0 * I)
    at /sandbox/petsc/petsc.next/src/vec/vec/interface/rvector.c:547
#4  0x00007f1a392e7110 in VecCreate_SeqCUDA (V=0x215c5e0)
    at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda.c:310
#5  0x00007f1a3932bb23 in VecSetType (vec=0x215c5e0, method=0x7f1a3ab28b0b "seqcuda")
    at /sandbox/petsc/petsc.next/src/vec/vec/interface/vecreg.c:51
#6  0x00007f1a39146894 in VecCreate_CUDA (v=0x215c5e0)
    at /sandbox/petsc/petsc.next/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu:190
#7  0x00007f1a3932bb23 in VecSetType (vec=0x215c5e0, method=0x21556b0 "cuda")
    at /sandbox/petsc/petsc.next/src/vec/vec/interface/vecreg.c:51
#8  0x00007f1a39f12dfe in DMCreateGlobalVector_DA (da=0x214ad10, g=0x7ffc8b95a5b8)
    at /sandbox/petsc/petsc.next/src/dm/impls/da/dadist.c:39
#9  0x00007f1a3a2deeca in DMCreateGlobalVector (dm=0x214ad10, vec=0x7ffc8b95a5b8)
    at /sandbox/petsc/petsc.next/src/dm/interface/dm.c:932
#10 0x0000000000401ebc in main (argc=7, argv=0x7ffc8b95a6d8)
---Type <return> to continue, or q <return> to quit---
    at /sandbox/petsc/petsc.next/src/ksp/ksp/examples/tests/ex32.c:48
(gdb) 
(gdb) up
#1  0x00007f1a3913ae48 in thrust::operator==<double, double> (x=..., y=...)
    at /usr/local/cuda/include/thrust/detail/complex/complex.inl:313
313	  return x.real() == y.real() && x.imag() == y.imag();
(gdb) p x
$1 = (const thrust::complex<double> &) @0x0: <error reading variable>
(gdb) p y
$2 = (const thrust::complex<double> &) @0x7ffc8b95a370: {data = {x = 0, y = 0}}
(gdb) up
#2  0x00007f1a3912f8ff in VecSet_SeqCUDA (xin=0x215c5e0, 
    alpha=<error reading variable: Cannot access memory at address 0x0>)
    at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda2.cu:775
775	  if (alpha == (PetscScalar)0.0) {
(gdb) p alpha
Cannot access memory at address 0x0
(gdb) up
#3  0x00007f1a39336ee7 in VecSet (x=0x215c5e0, alpha=0 + 0 * I)
    at /sandbox/petsc/petsc.next/src/vec/vec/interface/rvector.c:547
547	  ierr = (*x->ops->set)(x,alpha);CHKERRQ(ierr);
(gdb) p alpha



> On Feb 22, 2019, at 10:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  I am getting some crashes on our machine frog with complex cuda on those examples, though maybe not at the exact same line numbers.
> 
>  We need more help from our CUDA experts
> 
>   Barry
> 
> 
>> On Feb 22, 2019, at 8:33 AM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>> 
>> Sorry, I?ve tried both ex32 and ex39 in the src/ksp/ksp/examples/tests directory, both would give the same error.
>> I?ll try with valgrind or try another computer with a different GPU.
>> 
>> Thanks for confirming that complex *should* work on GPUs.
>> 
>> Randy M.
>> 
>> 
>> 
>>> On Feb 21, 2019, at 9:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>> 
>>> 
>>> Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and it runs though the -n1 n2 n3 options aren't used.
>>> 
>>> Barry
>>> 
>>> 
>>>> On Feb 21, 2019, at 6:20 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>>> 
>>>> Hi Barry and Satish,
>>>> 
>>>> Yes, sorry, I meant -dm_mat_type_aijcusparse?..
>>>> 
>>>> Here is an attempt to run ex39 under complex:
>>>> 
>>>> 0]PETSC ERROR: ------------------------------------------------------------------------
>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>> [0]PETSC ERROR: likely location of problem given in stack below
>>>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>>>> [0]PETSC ERROR:       is given.
>>>> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>>> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>>> [0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
>>>> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>>> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>>> [0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
>>>> [0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>> [0]PETSC ERROR: Signal received
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
>>>> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
>>>> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>> 
>>>> 
>>>> I used these options:
>>>> 
>>>> #!/bin/bash
>>>> 
>>>> export PETSC_ARCH=linux-gfortran-complex-debug
>>>> 
>>>> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
>>>> -ksp_type bcgs \
>>>> -ksp_rtol 1.e-6 \
>>>> -pc_type jacobi \
>>>> -ksp_monitor_true_residual \
>>>> -ksp_converged_reason \
>>>> -mat_type aijcusparse \
>>>> -vec_type cuda \
>>>> -n1 32 \
>>>> -n2 32 \
>>>> -n3 32 \
>>>> 
>>>> 
>>>> My next step was going to try valgrind and see if that turned something up.
>>>> 
>>>> Thanks, Randy
>>>> 
>>>> 
>>>>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>>> 
>>>>> 
>>>>> Randy,
>>>>> 
>>>>> Could you please cut and paste the entire error message you get. It worked for me. 
>>>>> 
>>>>> I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
>>>>> 
>>>>> Satish,
>>>>> 
>>>>>  I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>> 
>>>>>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>>>>>> We have installed PETSc following the instructions here:
>>>>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>>>>> 
>>>>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda 
>>>>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>>>>>> 
>>>>>> Is it possible to run PETSc on GPUS in complex mode?d
>>>>>> 
>>>>>> 
>>>>>> Thanks, Randy M.
>>>>> 
>>>> 
>>> 
>> 
> 


From bourdin at lsu.edu  Sat Feb 23 09:48:07 2019
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Sat, 23 Feb 2019 15:48:07 +0000
Subject: [petsc-users] Wrong Global Vector size when default section is
 built after dmdistribute?
Message-ID: <522344E6-DE01-4ED0-AE77-9295E3B6AD09@lsu.edu>

Hi,

My student Alex and I are trying to build an example combining natural to global ordering and constraints.
Constraints information do not seem to be distributed, so the most rational thing to do seems to be creating the default section after distribution, then figuring out a way to reconstruct the natural to global stuff.

In the example attached, we do this, then create a global vector, but for some reason that I do not understand, the global vector size is 0 (should be 18) when run on more than 1 CPU. If we make 2 calls to DMGetGlobalVector, we get a vector of the proper size. What are we doing wrong?

Regards,
Blaise


--
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/3a10a96d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.c
Type: application/octet-stream
Size: 2831 bytes
Desc: main.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/3a10a96d/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: twobytwo.exo
Type: application/octet-stream
Size: 1564 bytes
Desc: twobytwo.exo
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/3a10a96d/attachment-0001.obj>

From knepley at gmail.com  Sat Feb 23 14:39:17 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 23 Feb 2019 15:39:17 -0500
Subject: [petsc-users] Wrong Global Vector size when default section is
 built after dmdistribute?
In-Reply-To: <522344E6-DE01-4ED0-AE77-9295E3B6AD09@lsu.edu>
References: <522344E6-DE01-4ED0-AE77-9295E3B6AD09@lsu.edu>
Message-ID: <CAMYG4GmN31D-ZgrG3p=UWcTB1Pi6N0ghV5EUqU75JVphEkV7NA@mail.gmail.com>

On Sat, Feb 23, 2019 at 10:48 AM Blaise A Bourdin via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> My student Alex and I are trying to build an example combining natural to
> global ordering and constraints.
> Constraints information do not seem to be distributed,
>

Well dang. That is true. I never considered doing it the other way since
all my specifications come from compact things
like an FEM discretization and BC labels, which get distributed. But yes,
this is a big hole. I think it could be fixed without
too much work since the constraints are just another section+IS to
distribute, and the IS does not even have to be renumbered.
I think we can just call
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeFieldIS.html
on it.


> so the most rational thing to do seems to be creating the default section
> after distribution, then figuring out a way to reconstruct the natural to
> global stuff.
>

That is what I do, but we could do it the other way.


> In the example attached, we do this, then create a global vector, but for
> some reason that I do not understand, the global vector size is 0 (should
> be 18) when run on more than 1 CPU. If we make 2 calls to
> DMGetGlobalVector, we get a vector of the proper size. What are we doing
> wrong?
>

At the beginning, you turn on

  DMSetUseNatural(dm, PETSC_TRUE);

so during Distribute() it tries to make the mapping. This process creates a
global vector, which has size 0, because
the Section is empty. The reason it works when you call it again is that it
creates a new vector this time, now with
the new Section. So, to do it this way, don't set this flag at the
beginning. You have to setup the GlobalToNatural() afterwards.

Also, this shows that I should probably invalidate the global/local vector
cache after a new Section is set.

  Thanks,

    Matt


> Regards,
> Blaise
>
>
> --
> Department of Mathematics and Center for Computation & Technology
> Louisiana State University, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
> http://www.math.lsu.edu/~bourdin
>
>
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/949526e0/attachment.html>

From knepley at gmail.com  Sat Feb 23 15:09:17 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 23 Feb 2019 16:09:17 -0500
Subject: [petsc-users] Question with filedsplit in PETSc
In-Reply-To: <CY4PR11MB00569BEE94241D9905B1C8389C7E0@CY4PR11MB0056.namprd11.prod.outlook.com>
References: <CY4PR11MB00569BEE94241D9905B1C8389C7E0@CY4PR11MB0056.namprd11.prod.outlook.com>
Message-ID: <CAMYG4G=mqWLB4Log7RDEfC7C_tp6XQ8qa55OcVr6D_tk0Os87Q@mail.gmail.com>

On Thu, Feb 21, 2019 at 3:45 PM Zhu, Qiming via petsc-users <
petsc-users at mcs.anl.gov> wrote:

>
> Dear all,
>
>
> Sorry to disturb you. I am a user of Petsc. I am trying to use Fieldsplit
> in Petsc to do preconditioning for Navier-Stokes problem. I have some
> problems when I trying to use Fieldsplit function. I am now defining the
> nest matrix first, then I get the IS from nested matrix. But I find that my
> code just work for one core. When I change to parallel case, I could only
> get zero solution. I wonder is there any special requirements for IS
> definition in Fieldsplit? I include one code here. If you have any idea,
> hope you reply soon. Thank you for your help. Thank you very much.
>
I cleaned up the code a little so I could see what was going on. I attach
my version here. If you run on 1 proc, you get what you expect:

master *:~/Downloads/tmp/blaise$ $PETSC_DIR/../bin/mpiexec -n 1 ./ex5
-ksp_monitor_true_residual -ksp_view_mat -sol_view -rhs_view -sys_view

A00 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

A01 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A10 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A11 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0: (0, 5.)

row 1: (1, 6.)

row 2: (2, 7.)

row 3: (3, 8.)

IS Object: 1 MPI processes

  type: stride

Index set is permutation

Number of indices in (stride) set 4

0 0

1 1

2 2

3 3

IS Object: 1 MPI processes

  type: stride

Number of indices in (stride) set 4

0 4

1 5

2 6

3 7

  0 KSP preconditioned resid norm 2.828427124746e+00 true resid norm
1.428285685709e+01 ||r(i)||/||b|| 1.000000000000e+00

  1 KSP preconditioned resid norm 4.154074181055e-16 true resid norm
3.475547814546e-15 ||r(i)||/||b|| 2.433370192898e-16

Mat Object: 1 MPI processes

  type: nest

  Matrix object:

    type=nest, rows=2, cols=2

    MatNest structure:

    (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=4, cols=4

    (0,1) : type=mpiaij, rows=4, cols=4

    (1,0) : type=mpiaij, rows=4, cols=4

    (1,1) : prefix="fieldsplit_1_", type=mpiaij, rows=4, cols=4

Vec Object: Rhs 1 MPI processes

  type: mpi

Process [0]

1.

2.

3.

4.

5.

6.

7.

8.

Vec Object: Sol 1 MPI processes

  type: mpi

Process [0]

1.

1.

1.

1.

1.

1.

1.

1.

Mat Object: System Matrix 1 MPI processes

  type: seqaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

row 4: (4, 5.)

row 5: (5, 6.)

row 6: (6, 7.)
row 7: (7, 8.)

If you run on 2 procs, you get the "wrong" answer. This is because you
matrix is not in the order you think it is. I show this by converting to
AIJ and printing it. This happens because you are sticking together
_parallel_ matrices with Nest, so the local parts become contiguous:

master *:~/Downloads/tmp/blaise$ $PETSC_DIR/../bin/mpiexec -n 2 ./ex5
-ksp_monitor_true_residual -ksp_view_mat -sol_view -rhs_view -sys_view

A00 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

A01 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A10 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A11 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0: (0, 5.)

row 1: (1, 6.)

row 2: (2, 7.)

row 3: (3, 8.)

IS Object: 2 MPI processes

  type: stride

[0] Index set is permutation

[0] Number of indices in (stride) set 2

[0] 0 0

[0] 1 1

[1] Number of indices in (stride) set 2

[1] 0 4

[1] 1 5

IS Object: 2 MPI processes

  type: stride

[0] Number of indices in (stride) set 2

[0] 0 2

[0] 1 3

[1] Number of indices in (stride) set 2

[1] 0 6

[1] 1 7

  0 KSP preconditioned resid norm 3.135637450698e+00 true resid norm
1.428285685709e+01 ||r(i)||/||b|| 1.000000000000e+00

  1 KSP preconditioned resid norm -0.000000000000e+00 true resid norm
1.620317160370e-15 ||r(i)||/||b|| 1.134448924737e-16

Mat Object: 2 MPI processes

  type: nest

  Matrix object:

    type=nest, rows=2, cols=2

    MatNest structure:

    (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=4, cols=4

    (0,1) : type=mpiaij, rows=4, cols=4

    (1,0) : type=mpiaij, rows=4, cols=4

    (1,1) : prefix="fieldsplit_1_", type=mpiaij, rows=4, cols=4

Vec Object: Rhs 2 MPI processes

  type: mpi

Process [0]

1.

2.

3.

4.

Process [1]

5.

6.

7.

8.

Vec Object: Sol 2 MPI processes

  type: mpi

Process [0]

1.

1.

0.6

0.666667

Process [1]

1.66667

1.5

1.

1.

Mat Object: System Matrix 2 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 5.)

row 3: (3, 6.)

row 4: (4, 3.)

row 5: (5, 4.)

row 6: (6, 7.)
row 7: (7, 8.)

In general, I think its a bad idea to use Nest. Just build an AIJ matrix
the way you want and make some ISes.

  Thanks,

     Matt

> Yours sincerely,
>
> Qiming Zhu,
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/04892498/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5.c
Type: application/octet-stream
Size: 6185 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/04892498/attachment-0001.obj>

From qiming2 at illinois.edu  Sat Feb 23 16:29:01 2019
From: qiming2 at illinois.edu (Zhu, Qiming)
Date: Sat, 23 Feb 2019 22:29:01 +0000
Subject: [petsc-users] =?gb2312?b?tPC4tDogIFF1ZXN0aW9uIHdpdGggZmlsZWRz?=
 =?gb2312?b?cGxpdCBpbiBQRVRTYw==?=
In-Reply-To: <CAMYG4G=mqWLB4Log7RDEfC7C_tp6XQ8qa55OcVr6D_tk0Os87Q@mail.gmail.com>
References: <CY4PR11MB00569BEE94241D9905B1C8389C7E0@CY4PR11MB0056.namprd11.prod.outlook.com>,
	<CAMYG4G=mqWLB4Log7RDEfC7C_tp6XQ8qa55OcVr6D_tk0Os87Q@mail.gmail.com>
Message-ID: <CY4PR11MB0056301A43C2B494CC2FB5869C780@CY4PR11MB0056.namprd11.prod.outlook.com>

Dear Knepley,


Thank you for your detailed answer and help to this problem. I got your idea. And I also find it would be better to just use MPIAIJ matrix to construct this problem for fieldsplit. Thank you for your help all the time. Wish you have a nice day.


Yours sincerely,

Qiming Zhu

________________________________
???: Matthew Knepley <knepley at gmail.com>
????: 2019?2?23? 15:09:17
???: Zhu, Qiming
??: petsc-users at mcs.anl.gov
??: Re: [petsc-users] Question with filedsplit in PETSc

On Thu, Feb 21, 2019 at 3:45 PM Zhu, Qiming via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


Dear all,


Sorry to disturb you. I am a user of Petsc. I am trying to use Fieldsplit in Petsc to do preconditioning for Navier-Stokes problem. I have some problems when I trying to use Fieldsplit function. I am now defining the nest matrix first, then I get the IS from nested matrix. But I find that my code just work for one core. When I change to parallel case, I could only get zero solution. I wonder is there any special requirements for IS definition in Fieldsplit? I include one code here. If you have any idea, hope you reply soon. Thank you for your help. Thank you very much.

I cleaned up the code a little so I could see what was going on. I attach my version here. If you run on 1 proc, you get what you expect:


master *:~/Downloads/tmp/blaise$ $PETSC_DIR/../bin/mpiexec -n 1 ./ex5 -ksp_monitor_true_residual -ksp_view_mat -sol_view -rhs_view -sys_view

A00 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

A01 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A10 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A11 block print here.

Mat Object: 1 MPI processes

  type: mpiaij

row 0: (0, 5.)

row 1: (1, 6.)

row 2: (2, 7.)

row 3: (3, 8.)

IS Object: 1 MPI processes

  type: stride

Index set is permutation

Number of indices in (stride) set 4

0 0

1 1

2 2

3 3

IS Object: 1 MPI processes

  type: stride

Number of indices in (stride) set 4

0 4

1 5

2 6

3 7

  0 KSP preconditioned resid norm 2.828427124746e+00 true resid norm 1.428285685709e+01 ||r(i)||/||b|| 1.000000000000e+00

  1 KSP preconditioned resid norm 4.154074181055e-16 true resid norm 3.475547814546e-15 ||r(i)||/||b|| 2.433370192898e-16

Mat Object: 1 MPI processes

  type: nest

  Matrix object:

    type=nest, rows=2, cols=2

    MatNest structure:

    (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=4, cols=4

    (0,1) : type=mpiaij, rows=4, cols=4

    (1,0) : type=mpiaij, rows=4, cols=4

    (1,1) : prefix="fieldsplit_1_", type=mpiaij, rows=4, cols=4

Vec Object: Rhs 1 MPI processes

  type: mpi

Process [0]

1.

2.

3.

4.

5.

6.

7.

8.

Vec Object: Sol 1 MPI processes

  type: mpi

Process [0]

1.

1.

1.

1.

1.

1.

1.

1.

Mat Object: System Matrix 1 MPI processes

  type: seqaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

row 4: (4, 5.)

row 5: (5, 6.)

row 6: (6, 7.)

row 7: (7, 8.)

If you run on 2 procs, you get the "wrong" answer. This is because you matrix is not in the order you think it is. I show this by converting to AIJ and printing it. This happens because you are sticking together _parallel_ matrices with Nest, so the local parts become contiguous:


master *:~/Downloads/tmp/blaise$ $PETSC_DIR/../bin/mpiexec -n 2 ./ex5 -ksp_monitor_true_residual -ksp_view_mat -sol_view -rhs_view -sys_view

A00 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 3.)

row 3: (3, 4.)

A01 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A10 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0:

row 1:

row 2:

row 3:

A11 block print here.

Mat Object: 2 MPI processes

  type: mpiaij

row 0: (0, 5.)

row 1: (1, 6.)

row 2: (2, 7.)

row 3: (3, 8.)

IS Object: 2 MPI processes

  type: stride

[0] Index set is permutation

[0] Number of indices in (stride) set 2

[0] 0 0

[0] 1 1

[1] Number of indices in (stride) set 2

[1] 0 4

[1] 1 5

IS Object: 2 MPI processes

  type: stride

[0] Number of indices in (stride) set 2

[0] 0 2

[0] 1 3

[1] Number of indices in (stride) set 2

[1] 0 6

[1] 1 7

  0 KSP preconditioned resid norm 3.135637450698e+00 true resid norm 1.428285685709e+01 ||r(i)||/||b|| 1.000000000000e+00

  1 KSP preconditioned resid norm -0.000000000000e+00 true resid norm 1.620317160370e-15 ||r(i)||/||b|| 1.134448924737e-16

Mat Object: 2 MPI processes

  type: nest

  Matrix object:

    type=nest, rows=2, cols=2

    MatNest structure:

    (0,0) : prefix="fieldsplit_0_", type=mpiaij, rows=4, cols=4

    (0,1) : type=mpiaij, rows=4, cols=4

    (1,0) : type=mpiaij, rows=4, cols=4

    (1,1) : prefix="fieldsplit_1_", type=mpiaij, rows=4, cols=4

Vec Object: Rhs 2 MPI processes

  type: mpi

Process [0]

1.

2.

3.

4.

Process [1]

5.

6.

7.

8.

Vec Object: Sol 2 MPI processes

  type: mpi

Process [0]

1.

1.

0.6

0.666667

Process [1]

1.66667

1.5

1.

1.

Mat Object: System Matrix 2 MPI processes

  type: mpiaij

row 0: (0, 1.)

row 1: (1, 2.)

row 2: (2, 5.)

row 3: (3, 6.)

row 4: (4, 3.)

row 5: (5, 4.)

row 6: (6, 7.)

row 7: (7, 8.)

In general, I think its a bad idea to use Nest. Just build an AIJ matrix the way you want and make some ISes.

  Thanks,

     Matt

Yours sincerely,

Qiming Zhu,



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190223/cf49e12f/attachment-0001.html>

From ellen.price at cfa.harvard.edu  Sat Feb 23 17:47:23 2019
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Sat, 23 Feb 2019 18:47:23 -0500
Subject: [petsc-users] Using DMCOMPOSITE with TS
In-Reply-To: <CAOnA6qFX2vxMwM6xNxV_YBK1zjebZUPWsJZyu8wXetyzutFGdA@mail.gmail.com>
References: <CAOnA6qFX2vxMwM6xNxV_YBK1zjebZUPWsJZyu8wXetyzutFGdA@mail.gmail.com>
Message-ID: <8242a588-7365-e132-ac0c-1ea14e2ceb45@cfa.harvard.edu>

Quick update: I found that changing TS_EXACTFINALTIME_INTERPOLATE to
TS_EXACTFINALTIME_MATCHSTEP fixes the problem; is there a reason I can't
use the interpolation when using a DMCOMPOSITE?

Ellen Price


On 2/20/19 1:42 PM, Ellen Price wrote:
> Hi fellow PETSc users!
> 
> I am attempting to use a DMCOMPOSITE alongside TS and have run into some
> trouble. I'm attaching a MWE demonstrating the problem. The goal is to
> combine a DMDA3d for spatial data and a DMREDUNDANT for non-spatial,
> time-dependent fields. In the attached example, this additional field is
> temperature.
> 
> The DMDA data is currently just temperature-dependent, and the
> temperature is supposed to increase linearly. Unfortunately, no matter
> how I integrate (explicitly, using Euler or RK, or implicitly, using
> backward Euler or BDF), I get the wrong final temperature. The
> integration proceeds for 100 timesteps and then stops (verified by
> -ts_monitor). At a heating rate of 1, and an initial temperature of 150,
> I should get a final temperature of 250 very easily. However, I get
> something closer to 200 (not exact, which makes me think this isn't a
> simple missing-a-factor-of-2 error).
> 
> I'm currently using TSSetDM with the composite DM to let PETSc compute
> the Jacobian. Having checked all the inputs and outputs as well as I
> can, this is the only step that seems like it may be causing a problem;
> yet even explicit integration doesn't work, and that shouldn't need the
> Jacobian at all. I'm at a loss.
> 
> Run using:
> ./mwe -implicit yes -ts_type beuler -ts_monitor OR ./mwe -implicit no
> -ts_type euler -ts_monitor
> 
> Thanks in advance for any help,
> Ellen Price

From jed at jedbrown.org  Sat Feb 23 18:12:08 2019
From: jed at jedbrown.org (Jed Brown)
Date: Sat, 23 Feb 2019 17:12:08 -0700
Subject: [petsc-users] Using DMCOMPOSITE with TS
In-Reply-To: <8242a588-7365-e132-ac0c-1ea14e2ceb45@cfa.harvard.edu>
References: <CAOnA6qFX2vxMwM6xNxV_YBK1zjebZUPWsJZyu8wXetyzutFGdA@mail.gmail.com>
	<8242a588-7365-e132-ac0c-1ea14e2ceb45@cfa.harvard.edu>
Message-ID: <87lg26ujs7.fsf@jedbrown.org>

It shouldn't have any affect.  This will need to be debugged.  There's
no chance I'll have time for at least a week; hopefully one of the other
TS contributors can look sooner.

"Ellen M. Price via petsc-users" <petsc-users at mcs.anl.gov> writes:

> Quick update: I found that changing TS_EXACTFINALTIME_INTERPOLATE to
> TS_EXACTFINALTIME_MATCHSTEP fixes the problem; is there a reason I can't
> use the interpolation when using a DMCOMPOSITE?
>
> Ellen Price
>
>
> On 2/20/19 1:42 PM, Ellen Price wrote:
>> Hi fellow PETSc users!
>> 
>> I am attempting to use a DMCOMPOSITE alongside TS and have run into some
>> trouble. I'm attaching a MWE demonstrating the problem. The goal is to
>> combine a DMDA3d for spatial data and a DMREDUNDANT for non-spatial,
>> time-dependent fields. In the attached example, this additional field is
>> temperature.
>> 
>> The DMDA data is currently just temperature-dependent, and the
>> temperature is supposed to increase linearly. Unfortunately, no matter
>> how I integrate (explicitly, using Euler or RK, or implicitly, using
>> backward Euler or BDF), I get the wrong final temperature. The
>> integration proceeds for 100 timesteps and then stops (verified by
>> -ts_monitor). At a heating rate of 1, and an initial temperature of 150,
>> I should get a final temperature of 250 very easily. However, I get
>> something closer to 200 (not exact, which makes me think this isn't a
>> simple missing-a-factor-of-2 error).
>> 
>> I'm currently using TSSetDM with the composite DM to let PETSc compute
>> the Jacobian. Having checked all the inputs and outputs as well as I
>> can, this is the only step that seems like it may be causing a problem;
>> yet even explicit integration doesn't work, and that shouldn't need the
>> Jacobian at all. I'm at a loss.
>> 
>> Run using:
>> ./mwe -implicit yes -ts_type beuler -ts_monitor OR ./mwe -implicit no
>> -ts_type euler -ts_monitor
>> 
>> Thanks in advance for any help,
>> Ellen Price

From rupp at iue.tuwien.ac.at  Sun Feb 24 12:28:45 2019
From: rupp at iue.tuwien.ac.at (Karl Rupp)
Date: Sun, 24 Feb 2019 19:28:45 +0100
Subject: [petsc-users] GPUs, cud, complex
In-Reply-To: <4AF22B0C-DC48-452E-BD6C-934B9E62B5C4@mcs.anl.gov>
References: <087A8CF1-E16F-4359-B979-78B1F092A2B6@gmail.com>
	<10A298BF-CC26-4809-82A9-D1A62BF97285@anl.gov>
	<2AAE832D-91F6-435B-A9D1-B297F8F5EBD0@gmail.com>
	<61077234-D577-4AB6-AED2-E99D5BC0294C@mcs.anl.gov>
	<8ED67D2D-451C-42CA-84CE-27E6FEE7E0A1@gmail.com>
	<20733FC8-2437-4AC1-89D5-8CCB2E18397A@mcs.anl.gov>
	<4AF22B0C-DC48-452E-BD6C-934B9E62B5C4@mcs.anl.gov>
Message-ID: <502393d9-5125-4068-55e7-aec2055baee6@iue.tuwien.ac.at>

Hi,

just for information: we've seen some issues with Thrust in recent CUDA 
versions (mostly compilation issues). I don't know whether this is the 
cause of this particular error, though.

Best regards,
Karli



On 2/23/19 6:00 AM, Smith, Barry F. via petsc-users wrote:
> 
>     I get this in the debugger (can't make heads or tails out of it, does thrust handle complex in some way we don't understand so doesn't work with our source code?)
> 
> Thread 1 "ex32" received signal SIGSEGV, Segmentation fault.
> 0x00007f1a3913abd0 in thrust::complex<double>::real (this=0x0) at /usr/local/cuda/include/thrust/complex.h:324
> 324	  T real() const { return data.x; }
> (gdb) b
> Breakpoint 1 at 0x7f1a3913abd0: file /usr/local/cuda/include/thrust/complex.h, line 324.
> (gdb) bt
> #0  0x00007f1a3913abd0 in thrust::complex<double>::real (this=0x0)
>      at /usr/local/cuda/include/thrust/complex.h:324
> #1  0x00007f1a3913ae48 in thrust::operator==<double, double> (x=..., y=...)
>      at /usr/local/cuda/include/thrust/detail/complex/complex.inl:313
> #2  0x00007f1a3912f8ff in VecSet_SeqCUDA (xin=0x215c5e0,
>      alpha=<error reading variable: Cannot access memory at address 0x0>)
>      at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda2.cu:775
> #3  0x00007f1a39336ee7 in VecSet (x=0x215c5e0, alpha=0 + 0 * I)
>      at /sandbox/petsc/petsc.next/src/vec/vec/interface/rvector.c:547
> #4  0x00007f1a392e7110 in VecCreate_SeqCUDA (V=0x215c5e0)
>      at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda.c:310
> #5  0x00007f1a3932bb23 in VecSetType (vec=0x215c5e0, method=0x7f1a3ab28b0b "seqcuda")
>      at /sandbox/petsc/petsc.next/src/vec/vec/interface/vecreg.c:51
> #6  0x00007f1a39146894 in VecCreate_CUDA (v=0x215c5e0)
>      at /sandbox/petsc/petsc.next/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu:190
> #7  0x00007f1a3932bb23 in VecSetType (vec=0x215c5e0, method=0x21556b0 "cuda")
>      at /sandbox/petsc/petsc.next/src/vec/vec/interface/vecreg.c:51
> #8  0x00007f1a39f12dfe in DMCreateGlobalVector_DA (da=0x214ad10, g=0x7ffc8b95a5b8)
>      at /sandbox/petsc/petsc.next/src/dm/impls/da/dadist.c:39
> #9  0x00007f1a3a2deeca in DMCreateGlobalVector (dm=0x214ad10, vec=0x7ffc8b95a5b8)
>      at /sandbox/petsc/petsc.next/src/dm/interface/dm.c:932
> #10 0x0000000000401ebc in main (argc=7, argv=0x7ffc8b95a6d8)
> ---Type <return> to continue, or q <return> to quit---
>      at /sandbox/petsc/petsc.next/src/ksp/ksp/examples/tests/ex32.c:48
> (gdb)
> (gdb) up
> #1  0x00007f1a3913ae48 in thrust::operator==<double, double> (x=..., y=...)
>      at /usr/local/cuda/include/thrust/detail/complex/complex.inl:313
> 313	  return x.real() == y.real() && x.imag() == y.imag();
> (gdb) p x
> $1 = (const thrust::complex<double> &) @0x0: <error reading variable>
> (gdb) p y
> $2 = (const thrust::complex<double> &) @0x7ffc8b95a370: {data = {x = 0, y = 0}}
> (gdb) up
> #2  0x00007f1a3912f8ff in VecSet_SeqCUDA (xin=0x215c5e0,
>      alpha=<error reading variable: Cannot access memory at address 0x0>)
>      at /sandbox/petsc/petsc.next/src/vec/vec/impls/seq/seqcuda/veccuda2.cu:775
> 775	  if (alpha == (PetscScalar)0.0) {
> (gdb) p alpha
> Cannot access memory at address 0x0
> (gdb) up
> #3  0x00007f1a39336ee7 in VecSet (x=0x215c5e0, alpha=0 + 0 * I)
>      at /sandbox/petsc/petsc.next/src/vec/vec/interface/rvector.c:547
> 547	  ierr = (*x->ops->set)(x,alpha);CHKERRQ(ierr);
> (gdb) p alpha
> 
> 
> 
>> On Feb 22, 2019, at 10:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>
>>
>>   I am getting some crashes on our machine frog with complex cuda on those examples, though maybe not at the exact same line numbers.
>>
>>   We need more help from our CUDA experts
>>
>>    Barry
>>
>>
>>> On Feb 22, 2019, at 8:33 AM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>>
>>> Sorry, I?ve tried both ex32 and ex39 in the src/ksp/ksp/examples/tests directory, both would give the same error.
>>> I?ll try with valgrind or try another computer with a different GPU.
>>>
>>> Thanks for confirming that complex *should* work on GPUs.
>>>
>>> Randy M.
>>>
>>>
>>>
>>>> On Feb 21, 2019, at 9:53 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>>
>>>>
>>>> Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and it runs though the -n1 n2 n3 options aren't used.
>>>>
>>>> Barry
>>>>
>>>>
>>>>> On Feb 21, 2019, at 6:20 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
>>>>>
>>>>> Hi Barry and Satish,
>>>>>
>>>>> Yes, sorry, I meant -dm_mat_type_aijcusparse?..
>>>>>
>>>>> Here is an attempt to run ex39 under complex:
>>>>>
>>>>> 0]PETSC ERROR: ------------------------------------------------------------------------
>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>> [0]PETSC ERROR: likely location of problem given in stack below
>>>>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>>>>> [0]PETSC ERROR:       is given.
>>>>> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>>>> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
>>>>> [0]PETSC ERROR: [0] VecAYPX line 739 /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
>>>>> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>>>> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
>>>>> [0]PETSC ERROR: [0] KSPMonitor line 1714 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
>>>>> [0]PETSC ERROR: [0] KSPSolve line 678 /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>> [0]PETSC ERROR: Signal received
>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>>>>> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by root Thu Feb 21 19:03:37 2019
>>>>> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc --download-mpich=./mpich-3.3b1.tar.gz --download-fblaslapack=fblaslapack-3.4.2.tar.gz
>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>
>>>>>
>>>>> I used these options:
>>>>>
>>>>> #!/bin/bash
>>>>>
>>>>> export PETSC_ARCH=linux-gfortran-complex-debug
>>>>>
>>>>> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
>>>>> -ksp_type bcgs \
>>>>> -ksp_rtol 1.e-6 \
>>>>> -pc_type jacobi \
>>>>> -ksp_monitor_true_residual \
>>>>> -ksp_converged_reason \
>>>>> -mat_type aijcusparse \
>>>>> -vec_type cuda \
>>>>> -n1 32 \
>>>>> -n2 32 \
>>>>> -n3 32 \
>>>>>
>>>>>
>>>>> My next step was going to try valgrind and see if that turned something up.
>>>>>
>>>>> Thanks, Randy
>>>>>
>>>>>
>>>>>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>>>>
>>>>>>
>>>>>> Randy,
>>>>>>
>>>>>> Could you please cut and paste the entire error message you get. It worked for me.
>>>>>>
>>>>>> I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't exist).
>>>>>>
>>>>>> Satish,
>>>>>>
>>>>>>   I does appear we do not have a nightly test for cuda and complex, could that test be added to the nightly sweeps?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>> We are testing whether or not we can benefit from porting our PETSc code to use GPUS.
>>>>>>> We have installed PETSc following the instructions here:
>>>>>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>>>>>>
>>>>>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and -dm_mat_type aijcuda
>>>>>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out when using a COMPLEX version.
>>>>>>>
>>>>>>> Is it possible to run PETSc on GPUS in complex mode?d
>>>>>>>
>>>>>>>
>>>>>>> Thanks, Randy M.
>>>>>>
>>>>>
>>>>
>>>
>>
> 

From t.appel17 at imperial.ac.uk  Sun Feb 24 15:28:01 2019
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Sun, 24 Feb 2019 21:28:01 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
Message-ID: <A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>

Hi Matthew,

With the following data declaration and routines calls:

  DM :: da
  ISLocalToGlobalMapping :: ltog
  PetscInt, DIMENSION(:), POINTER :: id_ltog

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)


I get, with the most recent Intel Fortran compiler and a fresh ?git pull? PETsc:

forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is not associated with a target


Thibaut

On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I reckon that what corresponds best to my situation is to create my own MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself and fill my values with MatSetValues.

However, what is the correct way to call ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something missing in the documentation as my code just won't work.

Some examples include a PetscOffset argument which is not present in the documentation and there is no manual page for ISLocalToGlobalMappingGetIndicesF90: the only information I could find is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to allocate the index array yourself? I tried declaring it as a pointer too. I keep getting an error

1) There was a bug in that file, but I don't think it should have affected you. I will put in a PR

2)  You are supposed to pass in a pointer, unallocated. We manage all the allocation, which is why you call Restore. What happens when you do that?

  Thanks,

     Matt
[0]PETSC ERROR: #1 User provided function() line 0 in User file
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0

Minimal example:

PROGRAM test_dmda

#include <petsc/finclude/petscdm.h>
#include <petsc/finclude/petscdmda.h>

  USE PetscDMDA

  IMPLICIT NONE

  PetscErrorCode :: ierr
  PetscInt       :: irank, icx, icy, lx, ly, n_ltog
  DM             :: da
  ISLocalToGlobalMapping :: ltog

  PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
  !PetscInt, DIMENSION(:), POINTER :: id_ltog

  PetscInt, PARAMETER :: nx=24, ny=9, neq=4
  CHARACTER(LEN=100) :: err_msg

  CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)

  CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left corner is icx=', icx, &
                          'and icy=', icy, ' - the width is lx=', lx, ' and ly=', ly

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
  CHKERRA(ierr)

  ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)

  CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)

  CALL PetscFinalize(ierr)

END PROGRAM test_dmda



On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov> wrote:
Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I?ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.

However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory ?optimally?. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?

What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop on that in my preallocation and assembly routines.

You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.

  Thanks,

     Matt

Thanks very much,

Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190224/54459056/attachment-0001.html>

From knepley at gmail.com  Sun Feb 24 16:50:22 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 24 Feb 2019 17:50:22 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
Message-ID: <CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>

On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk>
wrote:

> Hi Matthew,
>
> With the following data declaration and routines calls:
>

I am not a Fortran90 expert, but your declaration looks different than the
ones I have seen work:


https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90

  Thanks,

     Matt


>
>   DM :: da
>   ISLocalToGlobalMapping :: ltog
>   PetscInt, DIMENSION(:), POINTER :: id_ltog
>
>   CALL
> DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),
> &
>
>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>   CHKERRA(ierr)
>
>   CALL DMSetUp(da,ierr)
>   CHKERRA(ierr)
>
>   CALL
> DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>   CHKERRA(ierr)
>
>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL DMDestroy(da,ierr)
>   CHKERRA(ierr)
>
>
> I get, with the most recent Intel Fortran compiler and a fresh ?git pull?
> PETsc:
>
> forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is
> not associated with a target
>
>
> Thibaut
>
> On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> I reckon that what corresponds best to my situation is to create my own
> MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself
> and fill my values with MatSetValues.
>
> However, what is the correct way to call ISLocalToGlobalMappingGetIndices
> in Fortran? It seems there's something missing in the documentation as my
> code just won't work.
>
> Some examples include a PetscOffset argument which is not present in the
> documentation and there is no manual page
> for ISLocalToGlobalMappingGetIndicesF90: the only information I could find
> is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to
> allocate the index array yourself? I tried declaring it as a pointer too. I
> keep getting an error
>
> 1) There was a bug in that file, but I don't think it should have affected
> you. I will put in a PR
>
> 2)  You are supposed to pass in a pointer, unallocated. We manage all the
> allocation, which is why you call Restore. What happens when you do that?
>
>   Thanks,
>
>      Matt
> [0]PETSC ERROR: #1 User provided function() line 0 in User file
> application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0
>
> Minimal example:
>
> PROGRAM test_dmda
>
> #include <petsc/finclude/petscdm.h>
> #include <petsc/finclude/petscdmda.h>
>
>   USE PetscDMDA
>
>   IMPLICIT NONE
>
>   PetscErrorCode :: ierr
>   PetscInt       :: irank, icx, icy, lx, ly, n_ltog
>   DM             :: da
>   ISLocalToGlobalMapping :: ltog
>
>   PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
>   !PetscInt, DIMENSION(:), POINTER :: id_ltog
>
>   PetscInt, PARAMETER :: nx=24, ny=9, neq=4
>   CHARACTER(LEN=100) :: err_msg
>
>   CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>
>   CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)
>
>
> CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
>
>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>   CHKERRA(ierr)
>
>   CALL DMSetUp(da,ierr)
>   CHKERRA(ierr)
>
>
> CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>   CHKERRA(ierr)
>
>   WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left
> corner is icx=', icx, &
>                           'and icy=', icy, ' - the width is lx=', lx, '
> and ly=', ly
>
>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
>   CHKERRA(ierr)
>
>   ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)
>
>   CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
>   CHKERRA(ierr)
>
>   DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)
>
>   CALL DMDestroy(da,ierr)
>   CHKERRA(ierr)
>
>   CALL PetscFinalize(ierr)
>
> END PROGRAM test_dmda
>
>
>
> On 21/02/2019 17:49, Matthew Knepley wrote:
>
> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> Dear PETSc developers/users,
>
> I?m solving linear PDEs on a regular grid with high-order finite
> differences, assembling an MPIAIJ matrix to solve linear systems or
> eigenvalue problems. I?ve been using vertex major, natural ordering for
> the parallelism with PetscSplitOwnership (yielding rectangular slices
> of the physical domain) and wanted to move to DMDA to have a more
> square-ish domain decomposition and minimize communication between
> processes.
>
> However, my application is memory critical, and I have finely-tuned
> matrix preallocation routines for allocating memory ?optimally?. It seems
> the memory of a DMDA matrix is allocated along the value of the stencil
> width of DMDACreate and the manual says about it
>
> ?These DMDA stencils have nothing directly to do with any finite
> difference stencils one might chose to use for a discretization?
>
> And despite reading the manual pages there must be something I do not
> understand in the DM topology, what is that "stencil width" for then? I
> will not use ghost values for my FD-method, right?
>
> What this is saying is, "You might be using some stencil that is not STAR
> or BOX, but we are preallocating according to one of those".
> If you really care about how much memory is preallocated, which it seems
> you do, then you might be able to use
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>
> to tell use exactly how to preallocate.
>
> I was then wondering if I could just create a MPIAIJ matrix, and with a
> PETSc routine get the global indices of the domain for each process:
> in other words, an equivalent of PetscSplitOwnership that gives me the DMDA
> unknown ordering. So I can feed and loop on that in my preallocation and
> assembly routines.
>
> You can make an MPIAIJ matrix yourself of course. It should have the same
> division of rows as the DMDA division of dofs. Also, MatSetValuesStencil()
> will not work for a custom matrix.
>
>   Thanks,
>
>      Matt
>
> Thanks very much,
>
> Thibaut
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190224/48611fb8/attachment.html>

From t.appel17 at imperial.ac.uk  Sun Feb 24 17:33:08 2019
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Sun, 24 Feb 2019 23:33:08 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
Message-ID: <069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>

Is that for the id_ltog argument? I tried declaring it as IS, and you can?t declare an deferred-shape array as target. Same message. XXX(:) is syntactic sugar for dimension(:) :: XXX

I?m really confused because this
https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
doesn?t use the xxxF90 variants of the routines we?re talking about?


Thibaut

On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Hi Matthew,

With the following data declaration and routines calls:

I am not a Fortran90 expert, but your declaration looks different than the ones I have seen work:

  https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90

  Thanks,

     Matt


  DM :: da
  ISLocalToGlobalMapping :: ltog
  PetscInt, DIMENSION(:), POINTER :: id_ltog

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)


I get, with the most recent Intel Fortran compiler and a fresh ?git pull? PETsc:

forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is not associated with a target


Thibaut

On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I reckon that what corresponds best to my situation is to create my own MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself and fill my values with MatSetValues.

However, what is the correct way to call ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something missing in the documentation as my code just won't work.

Some examples include a PetscOffset argument which is not present in the documentation and there is no manual page for ISLocalToGlobalMappingGetIndicesF90: the only information I could find is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to allocate the index array yourself? I tried declaring it as a pointer too. I keep getting an error

1) There was a bug in that file, but I don't think it should have affected you. I will put in a PR

2)  You are supposed to pass in a pointer, unallocated. We manage all the allocation, which is why you call Restore. What happens when you do that?

  Thanks,

     Matt
[0]PETSC ERROR: #1 User provided function() line 0 in User file
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0

Minimal example:

PROGRAM test_dmda

#include <petsc/finclude/petscdm.h>
#include <petsc/finclude/petscdmda.h>

  USE PetscDMDA

  IMPLICIT NONE

  PetscErrorCode :: ierr
  PetscInt       :: irank, icx, icy, lx, ly, n_ltog
  DM             :: da
  ISLocalToGlobalMapping :: ltog

  PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
  !PetscInt, DIMENSION(:), POINTER :: id_ltog

  PetscInt, PARAMETER :: nx=24, ny=9, neq=4
  CHARACTER(LEN=100) :: err_msg

  CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)

  CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left corner is icx=', icx, &
                          'and icy=', icy, ' - the width is lx=', lx, ' and ly=', ly

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
  CHKERRA(ierr)

  ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)

  CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)

  CALL PetscFinalize(ierr)

END PROGRAM test_dmda



On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I?ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.

However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory ?optimally?. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?

What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop on that in my preallocation and assembly routines.

You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.

  Thanks,

     Matt

Thanks very much,

Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190224/71e4b819/attachment-0001.html>

From knepley at gmail.com  Sun Feb 24 17:38:37 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 24 Feb 2019 18:38:37 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
	<069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>
Message-ID: <CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>

On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut <t.appel17 at imperial.ac.uk>
wrote:

> Is that for the id_ltog argument? I tried declaring it as IS, and you
> can?t declare an deferred-shape array as target. Same message. XXX(:) is
> syntactic sugar for dimension(:) :: XXX
>
> I?m really confused because this
>
> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
> doesn?t use the xxxF90 variants of the routines we?re talking about?
>

You can use the F77 stuff if you want. I thought you needed F90.  For F90,
I think you need to declare it

  PetscInt, pointer :: id_ltog(:)

like it is in the link I sent you.

  Thanks,

     Matt


> Thibaut
>
> On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk>
> wrote:
>
>> Hi Matthew,
>>
>> With the following data declaration and routines calls:
>>
>
> I am not a Fortran90 expert, but your declaration looks different than the
> ones I have seen work:
>
>
> https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90
>
>   Thanks,
>
>      Matt
>
>
>>
>>   DM :: da
>>   ISLocalToGlobalMapping :: ltog
>>   PetscInt, DIMENSION(:), POINTER :: id_ltog
>>
>>   CALL
>> DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),
>> &
>>
>>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL DMSetUp(da,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL
>> DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL DMDestroy(da,ierr)
>>   CHKERRA(ierr)
>>
>>
>> I get, with the most recent Intel Fortran compiler and a fresh ?git pull?
>> PETsc:
>>
>> forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it
>> is not associated with a target
>>
>>
>> Thibaut
>>
>> On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> I reckon that what corresponds best to my situation is to create my own
>> MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself
>> and fill my values with MatSetValues.
>>
>> However, what is the correct way to call ISLocalToGlobalMappingGetIndices
>> in Fortran? It seems there's something missing in the documentation as my
>> code just won't work.
>>
>> Some examples include a PetscOffset argument which is not present in the
>> documentation and there is no manual page
>> for ISLocalToGlobalMappingGetIndicesF90: the only information I could find
>> is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to
>> allocate the index array yourself? I tried declaring it as a pointer too. I
>> keep getting an error
>>
>> 1) There was a bug in that file, but I don't think it should have
>> affected you. I will put in a PR
>>
>> 2)  You are supposed to pass in a pointer, unallocated. We manage all the
>> allocation, which is why you call Restore. What happens when you do that?
>>
>>   Thanks,
>>
>>      Matt
>> [0]PETSC ERROR: #1 User provided function() line 0 in User file
>> application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0
>>
>> Minimal example:
>>
>> PROGRAM test_dmda
>>
>> #include <petsc/finclude/petscdm.h>
>> #include <petsc/finclude/petscdmda.h>
>>
>>   USE PetscDMDA
>>
>>   IMPLICIT NONE
>>
>>   PetscErrorCode :: ierr
>>   PetscInt       :: irank, icx, icy, lx, ly, n_ltog
>>   DM             :: da
>>   ISLocalToGlobalMapping :: ltog
>>
>>   PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
>>   !PetscInt, DIMENSION(:), POINTER :: id_ltog
>>
>>   PetscInt, PARAMETER :: nx=24, ny=9, neq=4
>>   CHARACTER(LEN=100) :: err_msg
>>
>>   CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>
>>   CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)
>>
>>
>> CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
>>
>>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL DMSetUp(da,ierr)
>>   CHKERRA(ierr)
>>
>>
>> CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>   CHKERRA(ierr)
>>
>>   WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left
>> corner is icx=', icx, &
>>                           'and icy=', icy, ' - the width is lx=', lx, '
>> and ly=', ly
>>
>>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)
>>
>>   CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
>>   CHKERRA(ierr)
>>
>>   DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)
>>
>>   CALL DMDestroy(da,ierr)
>>   CHKERRA(ierr)
>>
>>   CALL PetscFinalize(ierr)
>>
>> END PROGRAM test_dmda
>>
>>
>>
>> On 21/02/2019 17:49, Matthew Knepley wrote:
>>
>> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> Dear PETSc developers/users,
>>
>> I?m solving linear PDEs on a regular grid with high-order finite
>> differences, assembling an MPIAIJ matrix to solve linear systems or
>> eigenvalue problems. I?ve been using vertex major, natural ordering for
>> the parallelism with PetscSplitOwnership (yielding rectangular slices
>> of the physical domain) and wanted to move to DMDA to have a more
>> square-ish domain decomposition and minimize communication between
>> processes.
>>
>> However, my application is memory critical, and I have finely-tuned
>> matrix preallocation routines for allocating memory ?optimally?. It seems
>> the memory of a DMDA matrix is allocated along the value of the stencil
>> width of DMDACreate and the manual says about it
>>
>> ?These DMDA stencils have nothing directly to do with any finite
>> difference stencils one might chose to use for a discretization?
>>
>> And despite reading the manual pages there must be something I do not
>> understand in the DM topology, what is that "stencil width" for then? I
>> will not use ghost values for my FD-method, right?
>>
>> What this is saying is, "You might be using some stencil that is not STAR
>> or BOX, but we are preallocating according to one of those".
>> If you really care about how much memory is preallocated, which it seems
>> you do, then you might be able to use
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>>
>> to tell use exactly how to preallocate.
>>
>> I was then wondering if I could just create a MPIAIJ matrix, and with a
>> PETSc routine get the global indices of the domain for each process:
>> in other words, an equivalent of PetscSplitOwnership that gives me the DMDA
>> unknown ordering. So I can feed and loop on that in my preallocation and
>> assembly routines.
>>
>> You can make an MPIAIJ matrix yourself of course. It should have the same
>> division of rows as the DMDA division of dofs. Also, MatSetValuesStencil()
>> will not work for a custom matrix.
>>
>>   Thanks,
>>
>>      Matt
>>
>> Thanks very much,
>>
>> Thibaut
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190224/71d5961b/attachment.html>

From t.appel17 at imperial.ac.uk  Mon Feb 25 02:18:59 2019
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Mon, 25 Feb 2019 08:18:59 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
	<069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>,
	<CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>
Message-ID: <7ABDDB41-0417-42D0-8BF6-A7F74C7285FD@imperial.ac.uk>

Hi Matthew,

Yes I need F90 and the syntax in the file / yours
PetscInt, pointer :: id_ltog(:)

Is the exact same as
PetscInt, dimension(:), pointer :: id_ltog

They?re both modern fortran ?correct?

Anyways I tried both and I still get the same error message.

Thibaut

On 24 Feb 2019, at 23:38, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Is that for the id_ltog argument? I tried declaring it as IS, and you can?t declare an deferred-shape array as target. Same message. XXX(:) is syntactic sugar for dimension(:) :: XXX

I?m really confused because this
https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
doesn?t use the xxxF90 variants of the routines we?re talking about?

You can use the F77 stuff if you want. I thought you needed F90.  For F90, I think you need to declare it

  PetscInt, pointer :: id_ltog(:)

like it is in the link I sent you.

  Thanks,

     Matt

Thibaut

On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Hi Matthew,

With the following data declaration and routines calls:

I am not a Fortran90 expert, but your declaration looks different than the ones I have seen work:

  https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90

  Thanks,

     Matt


  DM :: da
  ISLocalToGlobalMapping :: ltog
  PetscInt, DIMENSION(:), POINTER :: id_ltog

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)


I get, with the most recent Intel Fortran compiler and a fresh ?git pull? PETsc:

forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is not associated with a target


Thibaut

On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I reckon that what corresponds best to my situation is to create my own MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself and fill my values with MatSetValues.

However, what is the correct way to call ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something missing in the documentation as my code just won't work.

Some examples include a PetscOffset argument which is not present in the documentation and there is no manual page for ISLocalToGlobalMappingGetIndicesF90: the only information I could find is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to allocate the index array yourself? I tried declaring it as a pointer too. I keep getting an error

1) There was a bug in that file, but I don't think it should have affected you. I will put in a PR

2)  You are supposed to pass in a pointer, unallocated. We manage all the allocation, which is why you call Restore. What happens when you do that?

  Thanks,

     Matt
[0]PETSC ERROR: #1 User provided function() line 0 in User file
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0

Minimal example:

PROGRAM test_dmda

#include <petsc/finclude/petscdm.h>
#include <petsc/finclude/petscdmda.h>

  USE PetscDMDA

  IMPLICIT NONE

  PetscErrorCode :: ierr
  PetscInt       :: irank, icx, icy, lx, ly, n_ltog
  DM             :: da
  ISLocalToGlobalMapping :: ltog

  PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
  !PetscInt, DIMENSION(:), POINTER :: id_ltog

  PetscInt, PARAMETER :: nx=24, ny=9, neq=4
  CHARACTER(LEN=100) :: err_msg

  CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)

  CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left corner is icx=', icx, &
                          'and icy=', icy, ' - the width is lx=', lx, ' and ly=', ly

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
  CHKERRA(ierr)

  ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)

  CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)

  CALL PetscFinalize(ierr)

END PROGRAM test_dmda



On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I?ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.

However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory ?optimally?. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?

What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop on that in my preallocation and assembly routines.

You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.

  Thanks,

     Matt

Thanks very much,

Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/71a8c17e/attachment-0001.html>

From eda.oktay at metu.edu.tr  Mon Feb 25 04:24:45 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Mon, 25 Feb 2019 13:24:45 +0300
Subject: [petsc-users] Finding nonzero indices
Message-ID: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>

Hello,

I am trying to find the indices (both row and column separately) of nonzero
entries of a sparse matrix in Petsc Binary Format. I found
MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
this is what I want or not.

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/136e43e4/attachment.html>

From knepley at gmail.com  Mon Feb 25 06:03:56 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 07:03:56 -0500
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <7ABDDB41-0417-42D0-8BF6-A7F74C7285FD@imperial.ac.uk>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
	<069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>
	<CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>
	<7ABDDB41-0417-42D0-8BF6-A7F74C7285FD@imperial.ac.uk>
Message-ID: <CAMYG4GkHUnRhbYLTwptwqmtjDkTqQfquJgby_+FSSKoCTHMJLQ@mail.gmail.com>

On Mon, Feb 25, 2019 at 3:19 AM Appel, Thibaut <t.appel17 at imperial.ac.uk>
wrote:

> Hi Matthew,
>
> Yes I need F90 and the syntax in the file / yours
>
> PetscInt, pointer :: id_ltog(:)
>
>
> Is the exact same as
>
> PetscInt, dimension(:), pointer :: id_ltog
>
>
> They?re both modern fortran ?correct?
>
> Anyways I tried both and I still get the same error message.
>

Okay, that file I sent the link to is tested every night, so I suspect a
compiler bug in your version.
Please send your entire example so I can run it here. Also, let us know
what Fortran compiler
and version you are using again.

  Thanks,

     Matt


> Thibaut
>
> On 24 Feb 2019, at 23:38, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut <t.appel17 at imperial.ac.uk>
> wrote:
>
>> Is that for the id_ltog argument? I tried declaring it as IS, and you
>> can?t declare an deferred-shape array as target. Same message. XXX(:) is
>> syntactic sugar for dimension(:) :: XXX
>>
>> I?m really confused because this
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
>> doesn?t use the xxxF90 variants of the routines we?re talking about?
>>
>
> You can use the F77 stuff if you want. I thought you needed F90.  For F90,
> I think you need to declare it
>
>   PetscInt, pointer :: id_ltog(:)
>
> like it is in the link I sent you.
>
>   Thanks,
>
>      Matt
>
>
>> Thibaut
>>
>> On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk>
>> wrote:
>>
>>> Hi Matthew,
>>>
>>> With the following data declaration and routines calls:
>>>
>>
>> I am not a Fortran90 expert, but your declaration looks different than
>> the ones I have seen work:
>>
>>
>> https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>>   DM :: da
>>>   ISLocalToGlobalMapping :: ltog
>>>   PetscInt, DIMENSION(:), POINTER :: id_ltog
>>>
>>>   CALL
>>> DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),
>>> &
>>>
>>>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL DMSetUp(da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL
>>> DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL DMDestroy(da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>
>>> I get, with the most recent Intel Fortran compiler and a fresh ?git
>>> pull? PETsc:
>>>
>>> forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it
>>> is not associated with a target
>>>
>>>
>>> Thibaut
>>>
>>> On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> I reckon that what corresponds best to my situation is to create my own
>>> MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself
>>> and fill my values with MatSetValues.
>>>
>>> However, what is the correct way to
>>> call ISLocalToGlobalMappingGetIndices in Fortran? It seems
>>> there's something missing in the documentation as my code just won't work.
>>>
>>> Some examples include a PetscOffset argument which is not present in the
>>> documentation and there is no manual page
>>> for ISLocalToGlobalMappingGetIndicesF90: the only information I could find
>>> is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to
>>> allocate the index array yourself? I tried declaring it as a pointer too. I
>>> keep getting an error
>>>
>>> 1) There was a bug in that file, but I don't think it should have
>>> affected you. I will put in a PR
>>>
>>> 2)  You are supposed to pass in a pointer, unallocated. We manage all
>>> the allocation, which is why you call Restore. What happens when you do
>>> that?
>>>
>>>   Thanks,
>>>
>>>      Matt
>>> [0]PETSC ERROR: #1 User provided function() line 0 in User file
>>> application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0
>>>
>>> Minimal example:
>>>
>>> PROGRAM test_dmda
>>>
>>> #include <petsc/finclude/petscdm.h>
>>> #include <petsc/finclude/petscdmda.h>
>>>
>>>   USE PetscDMDA
>>>
>>>   IMPLICIT NONE
>>>
>>>   PetscErrorCode :: ierr
>>>   PetscInt       :: irank, icx, icy, lx, ly, n_ltog
>>>   DM             :: da
>>>   ISLocalToGlobalMapping :: ltog
>>>
>>>   PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
>>>   !PetscInt, DIMENSION(:), POINTER :: id_ltog
>>>
>>>   PetscInt, PARAMETER :: nx=24, ny=9, neq=4
>>>   CHARACTER(LEN=100) :: err_msg
>>>
>>>   CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>
>>>   CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)
>>>
>>>
>>> CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
>>>
>>>   PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL DMSetUp(da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>
>>> CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left
>>> corner is icx=', icx, &
>>>                           'and icy=', icy, ' - the width is lx=', lx, '
>>> and ly=', ly
>>>
>>>   CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)
>>>
>>>   CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)
>>>
>>>   CALL DMDestroy(da,ierr)
>>>   CHKERRA(ierr)
>>>
>>>   CALL PetscFinalize(ierr)
>>>
>>> END PROGRAM test_dmda
>>>
>>>
>>>
>>> On 21/02/2019 17:49, Matthew Knepley wrote:
>>>
>>> On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> Dear PETSc developers/users,
>>>
>>> I?m solving linear PDEs on a regular grid with high-order finite
>>> differences, assembling an MPIAIJ matrix to solve linear systems or
>>> eigenvalue problems. I?ve been using vertex major, natural ordering for
>>> the parallelism with PetscSplitOwnership (yielding rectangular slices
>>> of the physical domain) and wanted to move to DMDA to have a more
>>> square-ish domain decomposition and minimize communication between
>>> processes.
>>>
>>> However, my application is memory critical, and I have finely-tuned
>>> matrix preallocation routines for allocating memory ?optimally?. It seems
>>> the memory of a DMDA matrix is allocated along the value of the stencil
>>> width of DMDACreate and the manual says about it
>>>
>>> ?These DMDA stencils have nothing directly to do with any finite
>>> difference stencils one might chose to use for a discretization?
>>>
>>> And despite reading the manual pages there must be something I do not
>>> understand in the DM topology, what is that "stencil width" for then? I
>>> will not use ghost values for my FD-method, right?
>>>
>>> What this is saying is, "You might be using some stencil that is not
>>> STAR or BOX, but we are preallocating according to one of those".
>>> If you really care about how much memory is preallocated, which it seems
>>> you do, then you might be able to use
>>>
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>>>
>>> to tell use exactly how to preallocate.
>>>
>>> I was then wondering if I could just create a MPIAIJ matrix, and with a
>>> PETSc routine get the global indices of the domain for each process:
>>> in other words, an equivalent of PetscSplitOwnership that gives me the DMDA
>>> unknown ordering. So I can feed and loop on that in my preallocation and
>>> assembly routines.
>>>
>>> You can make an MPIAIJ matrix yourself of course. It should have the
>>> same division of rows as the DMDA division of dofs. Also,
>>> MatSetValuesStencil() will not work for a custom matrix.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>> Thanks very much,
>>>
>>> Thibaut
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/e9fe04b6/attachment.html>

From t.appel17 at imperial.ac.uk  Mon Feb 25 06:15:19 2019
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Mon, 25 Feb 2019 12:15:19 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4GkHUnRhbYLTwptwqmtjDkTqQfquJgby_+FSSKoCTHMJLQ@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
	<069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>
	<CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>
	<7ABDDB41-0417-42D0-8BF6-A7F74C7285FD@imperial.ac.uk>
	<CAMYG4GkHUnRhbYLTwptwqmtjDkTqQfquJgby_+FSSKoCTHMJLQ@mail.gmail.com>
Message-ID: <b404a267-87b4-4144-05ad-11af43edbaf8@imperial.ac.uk>

Hi Matthew,

The file in the link you sent does not call DMGetLocalToGlobalMapping 
nor ISLocalToGlobalMappingGetIndicesF90 though?

Anyway, everything you wanted is there:

https://bitbucket.org/petsc/petsc/issues/262/fortran-unassociated-pointer-with

Thibaut


On 25/02/2019 12:03, Matthew Knepley wrote:
> On Mon, Feb 25, 2019 at 3:19 AM Appel, Thibaut 
> <t.appel17 at imperial.ac.uk <mailto:t.appel17 at imperial.ac.uk>> wrote:
>
>     Hi Matthew,
>
>     Yes I need F90 and the syntax in the file / yours
>>     PetscInt, pointer :: id_ltog(:)
>
>     Is the exact same as
>>     PetscInt, dimension(:), pointer :: id_ltog
>
>     They?re both modern fortran ?correct?
>
>     Anyways I tried both and I still get the same error message.
>
>
> Okay, that file I sent the link to is tested every night, so I suspect 
> a compiler bug in your version.
> Please send your entire example so I can run it here. Also, let us 
> know what Fortran compiler
> and version you are?using again.
>
> ? Thanks,
>
> ? ? ?Matt
>
>     Thibaut
>
>     On 24 Feb 2019, at 23:38, Matthew Knepley <knepley at gmail.com
>     <mailto:knepley at gmail.com>> wrote:
>
>>     On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut
>>     <t.appel17 at imperial.ac.uk <mailto:t.appel17 at imperial.ac.uk>> wrote:
>>
>>         Is that for the id_ltog argument? I tried declaring it as IS,
>>         and you can?t declare an deferred-shape array as target. Same
>>         message. XXX(:) is syntactic sugar for dimension(:) :: XXX
>>
>>         I?m really confused because this
>>         https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
>>         doesn?t use the xxxF90 variants of the routines we?re talking
>>         about?
>>
>>
>>     You can use the F77 stuff if you want. I thought you needed F90.?
>>     For F90, I think you need to declare it
>>
>>     ? PetscInt, pointer :: id_ltog(:)
>>
>>     like it is in the link I sent you.
>>
>>     ? Thanks,
>>
>>     ? ? ?Matt
>>
>>         Thibaut
>>
>>>         On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com
>>>         <mailto:knepley at gmail.com>> wrote:
>>>
>>>         On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut
>>>         <t.appel17 at imperial.ac.uk <mailto:t.appel17 at imperial.ac.uk>>
>>>         wrote:
>>>
>>>             Hi Matthew,
>>>
>>>             With the following data declaration and routines calls:
>>>
>>>
>>>         I am not a Fortran90 expert, but your declaration looks
>>>         different than the ones I have seen work:
>>>
>>>         https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90
>>>
>>>         ? Thanks,
>>>
>>>         ? ? ?Matt
>>>
>>>
>>>             ? DM :: da
>>>             ISLocalToGlobalMapping :: ltog
>>>             ??PetscInt, DIMENSION(:), POINTER :: id_ltog
>>>
>>>             ??CALL
>>>             DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),
>>>             &
>>>             ??PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL DMSetUp(da,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL
>>>             DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL
>>>             ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL
>>>             ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>             ??CALL DMDestroy(da,ierr)
>>>             ??CHKERRA(ierr)
>>>
>>>
>>>             I get, with the most recent Intel Fortran compiler and a
>>>             fresh ?git pull? PETsc:
>>>
>>>             forrtl: severe (408): fort: (7): Attempt to use pointer
>>>             ID_LTOG when it is not associated with a target
>>>
>>>
>>>             Thibaut
>>>
>>>>             On 22 Feb 2019, at 15:13, Matthew Knepley
>>>>             <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>
>>>>             On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via
>>>>             petsc-users <petsc-users at mcs.anl.gov
>>>>             <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>>             I reckon that what corresponds best to my situation is
>>>>             to create?my own MPIAIJ matrix, recover the DMDA local
>>>>             to global mapping,?preallocate myself and fill my
>>>>             values with MatSetValues.
>>>>
>>>>             However, what is the correct way to
>>>>             call?ISLocalToGlobalMappingGetIndices in Fortran? It
>>>>             seems there's?something missing in the documentation as
>>>>             my code just won't work.
>>>>
>>>>             Some examples include a PetscOffset argument which is
>>>>             not present?in the documentation and there is no manual
>>>>             page for?ISLocalToGlobalMappingGetIndicesF90: the only
>>>>             information I could?find is on
>>>>             /src/vec/is/utils/f90-custom/zisltogf90.c. Are
>>>>             you?supposed to allocate the index array yourself? I
>>>>             tried declaring?it as a pointer too. I keep getting an
>>>>             error
>>>>
>>>>             1) There was a bug in that file, but I don't think it
>>>>             should have affected you. I will put in a PR
>>>>
>>>>             2) ?You are supposed to pass in a pointer, unallocated.
>>>>             We manage all the allocation, which is why you call
>>>>             Restore. What happens when you do that?
>>>>
>>>>             ? Thanks,
>>>>
>>>>             ? ? ?Matt
>>>>             [0]PETSC ERROR: #1 User provided function() line 0 in
>>>>             User file
>>>>             application called MPI_Abort(MPI_COMM_SELF, -42924) -
>>>>             process 0
>>>>
>>>>             Minimal example:
>>>>
>>>>             PROGRAM test_dmda
>>>>
>>>>             #include <petsc/finclude/petscdm.h>
>>>>             #include <petsc/finclude/petscdmda.h>
>>>>
>>>>             ? USE PetscDMDA
>>>>
>>>>             ? IMPLICIT NONE
>>>>
>>>>             ? PetscErrorCode :: ierr
>>>>             ? PetscInt ? ? ? :: irank, icx, icy, lx, ly, n_ltog
>>>>             ? DM ? ? ? ? ? ? :: da
>>>>             ISLocalToGlobalMapping :: ltog
>>>>
>>>>             ? PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
>>>>             ? !PetscInt, DIMENSION(:), POINTER :: id_ltog
>>>>
>>>>             ? PetscInt, PARAMETER :: nx=24, ny=9, neq=4
>>>>             ? CHARACTER(LEN=100) :: err_msg
>>>>
>>>>             ? CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>>
>>>>             ? CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)
>>>>
>>>>             CALL?DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1),?&
>>>>             ??PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             ? CALL DMSetUp(da,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             CALL?DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my
>>>>             lower?left corner is icx=', icx, &
>>>>             ? ? ? 'and icy=', icy, ' - the width is?lx=', lx, ' and
>>>>             ly=', ly
>>>>
>>>>             ? CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             ? CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)
>>>>
>>>>             ? CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             ? CALL
>>>>             ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)
>>>>
>>>>             ? CALL DMDestroy(da,ierr)
>>>>             ? CHKERRA(ierr)
>>>>
>>>>             ? CALL PetscFinalize(ierr)
>>>>
>>>>             END PROGRAM test_dmda
>>>>
>>>>
>>>>
>>>>             On 21/02/2019 17:49, Matthew Knepley wrote:
>>>>>             On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel?via
>>>>>             petsc-users <petsc-users at mcs.anl.gov
>>>>>             <mailto:petsc-users at mcs.anl.gov>>?wrote:
>>>>>             Dear PETSc?developers/users,
>>>>>
>>>>>             I?m solving linear PDEs on a?regular grid with
>>>>>             high-order finite differences,?assembling an MPIAIJ
>>>>>             matrix to solve linear systems?or eigenvalue problems.
>>>>>             I?ve been using vertex?major, natural ordering for
>>>>>             the?parallelism with?PetscSplitOwnership (yielding
>>>>>             rectangular slices of?the physical domain) and wanted
>>>>>             to move to DMDA to?have a more square-ish domain
>>>>>             decomposition and?minimize communication?between
>>>>>             processes.
>>>>>
>>>>>             However, my application is?memory critical, and I have
>>>>>             finely-tuned matrix?preallocation routines for
>>>>>             allocating memory??optimally?. It seems the memory of
>>>>>             a DMDA matrix is?allocated along the value of
>>>>>             the?stencil width of?DMDACreate and the manual says
>>>>>             about it
>>>>>
>>>>>             ?These DMDA stencils?have nothing directly to do with
>>>>>             any finite difference?stencils one might chose to use
>>>>>             for a discretization?
>>>>>
>>>>>             And despite?reading the manual pages there must be
>>>>>             something I do?not understand in the DM topology, what
>>>>>             is that?"stencil width" for then? I will not use ghost
>>>>>             values?for my FD-method, right?
>>>>>
>>>>>             What this is saying is, "You might be using
>>>>>             some?stencil that is not STAR or BOX, but we are
>>>>>             preallocating?according to one of those".
>>>>>             If you really care about how much memory
>>>>>             is?preallocated, which it seems you do, then you might
>>>>>             be?able to use
>>>>>
>>>>>             https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html
>>>>>
>>>>>             to tell use exactly how to preallocate.
>>>>>
>>>>>             I was then wondering if I could just?create a MPIAIJ
>>>>>             matrix, and with a PETSc routine get?the global
>>>>>             indices of the domain for each process: in?other
>>>>>             words, an equivalent of PetscSplitOwnership
>>>>>             that?gives?me the DMDA unknown?ordering. So I can feed
>>>>>             and?loop on that in my preallocation and
>>>>>             assembly?routines.
>>>>>
>>>>>             You can make an MPIAIJ matrix yourself of course.
>>>>>             It?should have the same division of rows as the DMDA
>>>>>             division?of dofs. Also, MatSetValuesStencil() will not
>>>>>             work for a?custom matrix.
>>>>>
>>>>>             ? Thanks,
>>>>>
>>>>>             ? ? ?Matt
>>>>>
>>>>>             Thanks very much,
>>>>>
>>>>>             Thibaut
>>>>>
>>>>>
>>>>>             -- 
>>>>>             What most experimenters take for granted?before they
>>>>>             begin their experiments is?infinitely more interesting
>>>>>             than any results to?which their experiments lead.
>>>>>             -- Norbert Wiener
>>>>>
>>>>>             https://www.cse.buffalo.edu/~knepley/
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>>
>>>>             https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>>
>>>         https://www.cse.buffalo.edu/~knepley/
>>>         <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/9ab2c957/attachment-0001.html>

From mbuerkle at web.de  Mon Feb 25 08:00:47 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Mon, 25 Feb 2019 15:00:47 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
Message-ID: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/b5f3a5d6/attachment.html>

From knepley at gmail.com  Mon Feb 25 08:30:56 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 09:30:56 -0500
Subject: [petsc-users] Finding nonzero indices
In-Reply-To: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
References: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
Message-ID: <CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>

On Mon, Feb 25, 2019 at 5:25 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I am trying to find the indices (both row and column separately) of
> nonzero entries of a sparse matrix in Petsc Binary Format. I found
> MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
> this is what I want or not.
>

I would not do it that way, since it is fragile. I would write a loop like
this:

MAtGetOwnershipRange();
for (row = rStart; row < rEnd; ++row) {
  MatGetRow()
  PetscBinaryWrite()
  MatRestoreRow()
}

This will work for any matrix.

    Matt


> Best regards,
>
> Eda
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/8152b2c7/attachment.html>

From knepley at gmail.com  Mon Feb 25 08:36:12 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 09:36:12 -0500
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
Message-ID: <CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>

On Mon, Feb 25, 2019 at 9:00 AM Marius Buerkle via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> After commit 646531bb84 I get the following error message when
> calling MatLUFactorNumeric. It worked without a problem before.
>
> [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [6]PETSC ERROR: Nonconforming object sizes
> [6]PETSC ERROR: Non conforming matrix add: local sizes 51 x 54, 51 x 54
>

This seems impossible since it reports above that the sizes are the same,
but the error says they are different. Is it possible you source
got accidentally changed?

  Matt


> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [6]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1412-g646531bb84
>  GIT Date: 2019-02-07 21:47:41 +0300
> [6]PETSC ERROR:
> /home/marius/prog/ownstuff/fortran/programs/transomat_dev/transomat/transomat
> on a  named tono-hpc1 by marius Mon Feb 25 22:43:23 2019
> [6]PETSC ERROR: Configure options
> --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_opt_with_slepc
> --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpi
> cc --COPTFLAGS="-g -O0  -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0
>  -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0  -traceback " --with-mpi=1
> --with-x=0 --download-p
> armetis=1 --download-metis=1 --download-superlu_dist=1
> --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251
> --download-hwloc=1 --download-sowing=
> 1 --with-openmp=0 --with-pthread=0 --download-elemental=1
> --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff
> --with-cxx-dialect=c++11 --with-scala
> r-type=complex --with-debugging=1 --with-valgrind=0
> --with-blaslapack-lib="
> -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64
> -lmkl_scal
> apack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
> -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="
> -L/home/marius/intel/compilers
> _and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64
> -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64
> -liomp5 -lpthread -lm -
> ldl"
> --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
> --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.
> 1.144/linux/mkl
> --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
> --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_an
> d_libraries_2019.1.144/linux/mkl
> [6]PETSC ERROR: #1 MatAXPY() line 39 in
> /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
>
> best,
> Marus
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/ce5f9ee2/attachment.html>

From mbuerkle at web.de  Mon Feb 25 08:58:04 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Mon, 25 Feb 2019 15:58:04 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
Message-ID: <trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/89734958/attachment.html>

From stefano.zampini at gmail.com  Mon Feb 25 09:05:21 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 25 Feb 2019 07:05:21 -0800
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
Message-ID: <CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>

I don't see any issue with the commit.
Did you recompile petsc ? Can you send us a MWE to reproduce the error?
Have you checked under valgrind?

Il Lun 25 Feb 2019, 06:58 Marius Buerkle via petsc-users <
petsc-users at mcs.anl.gov> ha scritto:

> No excatly same source on my side. It works with commits prior to
> 646531bb84  and fails starting from 646531bb84.
>
>
>
> On Mon, Feb 25, 2019 at 9:00 AM Marius Buerkle via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> After commit 646531bb84 I get the following error message when
>> calling MatLUFactorNumeric. It worked without a problem before.
>>
>> [6]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [6]PETSC ERROR: Nonconforming object sizes
>> [6]PETSC ERROR: Non conforming matrix add: local sizes 51 x 54, 51 x 54
>>
>
> This seems impossible since it reports above that the sizes are the same,
> but the error says they are different. Is it possible you source
> got accidentally changed?
>
>   Matt
>
>
>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [6]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1412-g646531bb84
>>  GIT Date: 2019-02-07 21:47:41 +0300
>> [6]PETSC ERROR:
>> /home/marius/prog/ownstuff/fortran/programs/transomat_dev/transomat/transomat
>> on a  named tono-hpc1 by marius Mon Feb 25 22:43:23 2019
>> [6]PETSC ERROR: Configure options
>> --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_opt_with_slepc
>> --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpi
>> cc --COPTFLAGS="-g -O0  -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0
>>  -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0  -traceback " --with-mpi=1
>> --with-x=0 --download-p
>> armetis=1 --download-metis=1 --download-superlu_dist=1
>> --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251
>> --download-hwloc=1 --download-sowing=
>> 1 --with-openmp=0 --with-pthread=0 --download-elemental=1
>> --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff
>> --with-cxx-dialect=c++11 --with-scala
>> r-type=complex --with-debugging=1 --with-valgrind=0
>> --with-blaslapack-lib="
>> -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64
>> -lmkl_scal
>> apack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
>> -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="
>> -L/home/marius/intel/compilers
>> _and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64
>> -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64
>> -liomp5 -lpthread -lm -
>> ldl"
>> --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
>> --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.
>> 1.144/linux/mkl
>> --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
>> --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_an
>> d_libraries_2019.1.144/linux/mkl
>> [6]PETSC ERROR: #1 MatAXPY() line 39 in
>> /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
>>
>> best,
>> Marus
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/54019787/attachment-0001.html>

From eda.oktay at metu.edu.tr  Mon Feb 25 09:09:44 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Mon, 25 Feb 2019 18:09:44 +0300
Subject: [petsc-users] Finding nonzero indices
In-Reply-To: <CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>
References: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
	<CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>
Message-ID: <CA+vu_XiFCJ+FZFLqzXDs5G7g6hXXfOyjNSLXOLaqm1F6Zk1u8Q@mail.gmail.com>

>
> Thank you for your answer Matt, but I have some questions about
> PetscBinaryWrite() since I am a new user.
>
> Since I am going to use this function, I guess I have to open a new binary
> file with PetscViewerBinaryOpen
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html#PetscViewerBinaryOpen>
>  and  PetscViewerBinaryGetDescriptor
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryGetDescriptor.html#PetscViewerBinaryGetDescriptor>,
> and destroy viewer by  PetscViewerDestroy
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerDestroy.html#PetscViewerDestroy>.
> Do I write these functions into for loop or not?
>
> Eda
>
> Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 17:31 tarihinde
> ?unu yazd?:
>
>> On Mon, Feb 25, 2019 at 5:25 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hello,
>>>
>>> I am trying to find the indices (both row and column separately) of
>>> nonzero entries of a sparse matrix in Petsc Binary Format. I found
>>> MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
>>> this is what I want or not.
>>>
>>
>> I would not do it that way, since it is fragile. I would write a loop
>> like this:
>>
>> MAtGetOwnershipRange();
>> for (row = rStart; row < rEnd; ++row) {
>>   MatGetRow()
>>   PetscBinaryWrite()
>>   MatRestoreRow()
>> }
>>
>> This will work for any matrix.
>>
>>     Matt
>>
>>
>>> Best regards,
>>>
>>> Eda
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/31f846c9/attachment.html>

From knepley at gmail.com  Mon Feb 25 09:28:54 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 10:28:54 -0500
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
Message-ID: <CAMYG4GmMv+onjYEZ-mMiTRVp0hzi=oDbOCnxK7FQaPxKJ2ia6g@mail.gmail.com>

On Mon, Feb 25, 2019 at 9:58 AM Marius Buerkle <mbuerkle at web.de> wrote:

> No excatly same source on my side. It works with commits prior to
> 646531bb84  and fails starting from 646531bb84.
>

Can you run in the debugger? As I said, this cannot be understood without
something very strange happening, like a memory
overwrite.

  Thanks,

    Matt


>
>
> On Mon, Feb 25, 2019 at 9:00 AM Marius Buerkle via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi,
>>
>> After commit 646531bb84 I get the following error message when
>> calling MatLUFactorNumeric. It worked without a problem before.
>>
>> [6]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [6]PETSC ERROR: Nonconforming object sizes
>> [6]PETSC ERROR: Non conforming matrix add: local sizes 51 x 54, 51 x 54
>>
>
> This seems impossible since it reports above that the sizes are the same,
> but the error says they are different. Is it possible you source
> got accidentally changed?
>
>   Matt
>
>
>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [6]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1412-g646531bb84
>>  GIT Date: 2019-02-07 21:47:41 +0300
>> [6]PETSC ERROR:
>> /home/marius/prog/ownstuff/fortran/programs/transomat_dev/transomat/transomat
>> on a  named tono-hpc1 by marius Mon Feb 25 22:43:23 2019
>> [6]PETSC ERROR: Configure options
>> --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_opt_with_slepc
>> --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpi
>> cc --COPTFLAGS="-g -O0  -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0
>>  -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0  -traceback " --with-mpi=1
>> --with-x=0 --download-p
>> armetis=1 --download-metis=1 --download-superlu_dist=1
>> --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251
>> --download-hwloc=1 --download-sowing=
>> 1 --with-openmp=0 --with-pthread=0 --download-elemental=1
>> --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff
>> --with-cxx-dialect=c++11 --with-scala
>> r-type=complex --with-debugging=1 --with-valgrind=0
>> --with-blaslapack-lib="
>> -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64
>> -lmkl_scal
>> apack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
>> -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib="
>> -L/home/marius/intel/compilers
>> _and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64
>> -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64
>> -liomp5 -lpthread -lm -
>> ldl"
>> --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
>> --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.
>> 1.144/linux/mkl
>> --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl
>> --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_an
>> d_libraries_2019.1.144/linux/mkl
>> [6]PETSC ERROR: #1 MatAXPY() line 39 in
>> /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
>>
>> best,
>> Marus
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/9e0dfdb2/attachment.html>

From knepley at gmail.com  Mon Feb 25 09:29:24 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 10:29:24 -0500
Subject: [petsc-users] Finding nonzero indices
In-Reply-To: <CA+vu_XiFCJ+FZFLqzXDs5G7g6hXXfOyjNSLXOLaqm1F6Zk1u8Q@mail.gmail.com>
References: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
	<CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>
	<CA+vu_XiFCJ+FZFLqzXDs5G7g6hXXfOyjNSLXOLaqm1F6Zk1u8Q@mail.gmail.com>
Message-ID: <CAMYG4Gk55rD9yG2o_=03N=485rGSsQMXpU=JsBJO3O8MByOxfQ@mail.gmail.com>

On Mon, Feb 25, 2019 at 10:20 AM Eda Oktay <eda.oktay at metu.edu.tr> wrote:

> Thank you for your answer Matt, but I have some questions about
>> PetscBinaryWrite() since I am a new user.
>>
>> Since I am going to use this function, I guess I have to open a new
>> binary file with PetscViewerBinaryOpen
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html#PetscViewerBinaryOpen>
>>  and  PetscViewerBinaryGetDescriptor
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryGetDescriptor.html#PetscViewerBinaryGetDescriptor>,
>> and destroy viewer by  PetscViewerDestroy
>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerDestroy.html#PetscViewerDestroy>.
>> Do I write these functions into for loop or not?
>>
>
No, they go outside.

  Thanks,

    Matt


> Eda
>>
>> Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 17:31 tarihinde
>> ?unu yazd?:
>>
>>> On Mon, Feb 25, 2019 at 5:25 AM Eda Oktay via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am trying to find the indices (both row and column separately) of
>>>> nonzero entries of a sparse matrix in Petsc Binary Format. I found
>>>> MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
>>>> this is what I want or not.
>>>>
>>>
>>> I would not do it that way, since it is fragile. I would write a loop
>>> like this:
>>>
>>> MAtGetOwnershipRange();
>>> for (row = rStart; row < rEnd; ++row) {
>>>   MatGetRow()
>>>   PetscBinaryWrite()
>>>   MatRestoreRow()
>>> }
>>>
>>> This will work for any matrix.
>>>
>>>     Matt
>>>
>>>
>>>> Best regards,
>>>>
>>>> Eda
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/4b11113c/attachment.html>

From mbuerkle at web.de  Mon Feb 25 09:46:12 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Mon, 25 Feb 2019 16:46:12 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
Message-ID: <trinity-4582e5a4-f266-46e1-86d8-b99753c76252-1551109572370@3c-app-webde-bap17>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/edba57f7/attachment-0001.html>

From bourdin at lsu.edu  Mon Feb 25 13:38:34 2019
From: bourdin at lsu.edu (Blaise A Bourdin)
Date: Mon, 25 Feb 2019 19:38:34 +0000
Subject: [petsc-users] Wrong Global Vector size when default section is
 built after dmdistribute?
In-Reply-To: <CAMYG4GmN31D-ZgrG3p=UWcTB1Pi6N0ghV5EUqU75JVphEkV7NA@mail.gmail.com>
References: <522344E6-DE01-4ED0-AE77-9295E3B6AD09@lsu.edu>
	<CAMYG4GmN31D-ZgrG3p=UWcTB1Pi6N0ghV5EUqU75JVphEkV7NA@mail.gmail.com>
Message-ID: <6DFCF871-9D4F-46E0-BDC2-3167C8A2B179@lsu.edu>

Of course, you are right.
In retrospect, trying to create the natural to global SF during distribution does not seem to be the smartest idea?
Would you have time to take a pass at distributing the constrains?
On our side, we will work on patch that separates the creation of the Naturaltoglobal SF from distribution, and works when the default section was created on the sequential or distributed DM.
It would just be noce to have both cases working

Regards,
Blaise



On Feb 23, 2019, at 2:39 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sat, Feb 23, 2019 at 10:48 AM Blaise A Bourdin via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi,

My student Alex and I are trying to build an example combining natural to global ordering and constraints.
Constraints information do not seem to be distributed,

Well dang. That is true. I never considered doing it the other way since all my specifications come from compact things
like an FEM discretization and BC labels, which get distributed. But yes, this is a big hole. I think it could be fixed without
too much work since the constraints are just another section+IS to distribute, and the IS does not even have to be renumbered.
I think we can just call https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeFieldIS.html on it.

so the most rational thing to do seems to be creating the default section after distribution, then figuring out a way to reconstruct the natural to global stuff.

That is what I do, but we could do it the other way.

In the example attached, we do this, then create a global vector, but for some reason that I do not understand, the global vector size is 0 (should be 18) when run on more than 1 CPU. If we make 2 calls to DMGetGlobalVector, we get a vector of the proper size. What are we doing wrong?

At the beginning, you turn on

  DMSetUseNatural(dm, PETSC_TRUE);

so during Distribute() it tries to make the mapping. This process creates a global vector, which has size 0, because
the Section is empty. The reason it works when you call it again is that it creates a new vector this time, now with
the new Section. So, to do it this way, don't set this flag at the beginning. You have to setup the GlobalToNatural() afterwards.

Also, this shows that I should probably invalidate the global/local vector cache after a new Section is set.

  Thanks,

    Matt

Regards,
Blaise


--
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin









--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

--
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/5ddaa10f/attachment.html>

From yyang85 at stanford.edu  Mon Feb 25 15:32:55 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Mon, 25 Feb 2019 21:32:55 +0000
Subject: [petsc-users] Kronecker product
Message-ID: <DM6PR02MB52427C1B26F932A5B00442CFD27A0@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I'd like to ask whether PETSc has a function to compute the Kronecker product of a sparse matrix with an identity matrix? A Google search didn't lead me to a manual page (like most of the other PETSc functions), so I'm wondering if this has been implemented yet.

Thanks very much!

Best,
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/e2b7d462/attachment-0001.html>

From knepley at gmail.com  Mon Feb 25 15:39:02 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 16:39:02 -0500
Subject: [petsc-users] Kronecker product
In-Reply-To: <DM6PR02MB52427C1B26F932A5B00442CFD27A0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427C1B26F932A5B00442CFD27A0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4GmM3k7=gHWF=RpmZppVXpC6Ps+CyaK8x11YEYAkYe5aJg@mail.gmail.com>

On Mon, Feb 25, 2019 at 4:33 PM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello team,
>
>
>
> I?d like to ask whether PETSc has a function to compute the Kronecker
> product of a sparse matrix with an identity matrix? A Google search didn?t
> lead me to a manual page (like most of the other PETSc functions), so I?m
> wondering if this has been implemented yet.
>

It has not been released, but is in a branch that Richard Mills is working
on. We hope to have this in master very soon.

  Thanks,

    Matt


>
>
> Thanks very much!
>
>
>
> Best,
>
> Yuyun
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/7c21e00a/attachment.html>

From jed at jedbrown.org  Mon Feb 25 15:41:57 2019
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 25 Feb 2019 14:41:57 -0700
Subject: [petsc-users] Kronecker product
In-Reply-To: <DM6PR02MB52427C1B26F932A5B00442CFD27A0@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427C1B26F932A5B00442CFD27A0@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <87wolnsfyy.fsf@jedbrown.org>

MatCreateMAIJ does that (implicitly).

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMAIJ.html

If you want a Kronecker product with a non-identity matrix, this PR may
be of interest.

https://bitbucket.org/petsc/petsc/pull-requests/1334/rmills-mat-kaij/diff

Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov> writes:

> Hello team,
>
> I'd like to ask whether PETSc has a function to compute the Kronecker product of a sparse matrix with an identity matrix? A Google search didn't lead me to a manual page (like most of the other PETSc functions), so I'm wondering if this has been implemented yet.
>
> Thanks very much!
>
> Best,
> Yuyun

From huq2090 at gmail.com  Mon Feb 25 15:55:21 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Mon, 25 Feb 2019 15:55:21 -0600
Subject: [petsc-users] Error Norm_2
Message-ID: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>

Hello PETSc Developers,

I have solved a very simple poisson problem with different matrix sizes (10
to 10^7).
But when I have compared error norm_2 for the solution, I got the attached
curve.
It looks like error norm_2 increases with increasing matrix size. Shouldn't
it decrease rather with increasing matrix size?

Thanks.

Sincerely,
Huq
-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/bac3b01f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Error norm_2_withoutJacobi.png
Type: image/png
Size: 25278 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/bac3b01f/attachment-0001.png>

From knepley at gmail.com  Mon Feb 25 16:06:22 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 Feb 2019 17:06:22 -0500
Subject: [petsc-users] Error Norm_2
In-Reply-To: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>
References: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>
Message-ID: <CAMYG4Gmcj3FZ311esm22Uk6Q6ezmpgW1TToMSn1RZgPqePBjtA@mail.gmail.com>

On Mon, Feb 25, 2019 at 4:56 PM Fazlul Huq via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello PETSc Developers,
>
> I have solved a very simple poisson problem with different matrix sizes
> (10 to 10^7).
> But when I have compared error norm_2 for the solution, I got the attached
> curve.
> It looks like error norm_2 increases with increasing matrix size.
> Shouldn't it decrease rather with increasing matrix size?
>

Yes that is completely wrong.

  Matt


> Thanks.
>
> Sincerely,
> Huq
> --
>
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/94ad4128/attachment.html>

From bsmith at mcs.anl.gov  Mon Feb 25 17:15:40 2019
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 25 Feb 2019 23:15:40 +0000
Subject: [petsc-users] Error Norm_2
In-Reply-To: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>
References: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>
Message-ID: <11F74DFF-34F4-4179-A3EB-6306730199CE@anl.gov>


   How are you computing the error norm? 

   You need to use the L2 norm in the computations, not the l2 norm. 

   Also you need to make sure the convergence criteria you use for the algebraic system is smaller than the descritzation error 


   Barry


> On Feb 25, 2019, at 1:55 PM, Fazlul Huq via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hello PETSc Developers,
> 
> I have solved a very simple poisson problem with different matrix sizes (10 to 10^7).
> But when I have compared error norm_2 for the solution, I got the attached curve.
> It looks like error norm_2 increases with increasing matrix size. Shouldn't it decrease rather with increasing matrix size?
> 
> Thanks.
> 
> Sincerely,
> Huq
> -- 
> 
> Fazlul Huq
> Graduate Research Assistant
> Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> University of Illinois at Urbana-Champaign (UIUC)
> E-mail: huq2090 at gmail.com
> <Error norm_2_withoutJacobi.png>


From huq2090 at gmail.com  Mon Feb 25 20:37:03 2019
From: huq2090 at gmail.com (Fazlul Huq)
Date: Mon, 25 Feb 2019 20:37:03 -0600
Subject: [petsc-users] Error Norm_2
In-Reply-To: <11F74DFF-34F4-4179-A3EB-6306730199CE@anl.gov>
References: <CAE5QvP105Ad2H7YABN3qdp-gNpdEXjFRQP5vkvhd=T1VUXCTzw@mail.gmail.com>
	<11F74DFF-34F4-4179-A3EB-6306730199CE@anl.gov>
Message-ID: <CAE5QvP1GvMXaRM59tD3MvhkffpHggEtFYnpRTAf3=1fnBQfahg@mail.gmail.com>

Hello PETSc Developers,

Thanks for the response!

To calculate error, I first calculate the analytical solution and put it
inside vector s.
Then I took difference between analytical solution and numerical solution
and put it inside vector x.
Then I calculate the NORM_2 of x.
The code is as follows:
/*
     Check the error
  */
  for (i = 0; i < n; i++)
    {
      k1 = (float) (i+1)/(n+1);
      k2 = -0.5 * k1 * k1 + 5.5 * k1 + 10;
      ierr = VecSetValues(s, 1, &i, &k2, INSERT_VALUES);CHKERRQ(ierr);
    }
  ierr = VecAXPY(x,-1.0,s);CHKERRQ(ierr);
  ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr);
  if (norm > tol) {
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Second Norm of error
%g\n",(double)norm);CHKERRQ(ierr);
  }

Thanks again.
Sincerely,
Huq

On Mon, Feb 25, 2019 at 5:15 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>    How are you computing the error norm?
>
>    You need to use the L2 norm in the computations, not the l2 norm.
>
>    Also you need to make sure the convergence criteria you use for the
> algebraic system is smaller than the descritzation error
>
>
>    Barry
>
>
> > On Feb 25, 2019, at 1:55 PM, Fazlul Huq via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
> >
> > Hello PETSc Developers,
> >
> > I have solved a very simple poisson problem with different matrix sizes
> (10 to 10^7).
> > But when I have compared error norm_2 for the solution, I got the
> attached curve.
> > It looks like error norm_2 increases with increasing matrix size.
> Shouldn't it decrease rather with increasing matrix size?
> >
> > Thanks.
> >
> > Sincerely,
> > Huq
> > --
> >
> > Fazlul Huq
> > Graduate Research Assistant
> > Department of Nuclear, Plasma & Radiological Engineering (NPRE)
> > University of Illinois at Urbana-Champaign (UIUC)
> > E-mail: huq2090 at gmail.com
> > <Error norm_2_withoutJacobi.png>
>
>

-- 

Fazlul Huq
Graduate Research Assistant
Department of Nuclear, Plasma & Radiological Engineering (NPRE)
University of Illinois at Urbana-Champaign (UIUC)
E-mail: huq2090 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/1316421c/attachment.html>

From mbuerkle at web.de  Mon Feb 25 23:51:42 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Tue, 26 Feb 2019 06:51:42 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
Message-ID: <trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190226/78b09cc1/attachment.html>

From stefano.zampini at gmail.com  Tue Feb 26 00:08:05 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Mon, 25 Feb 2019 22:08:05 -0800
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
Message-ID: <59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>

We need a stacktrace or an MWE to understand what is going on

> On Feb 25, 2019, at 9:51 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> 
> I checked the problem further. I am not sure if this is the cause but what happens is that I use MatAXPY with Y = MATELEMENTAL and X= MATMPIAIJ before calling MatLUFactorNumeric, it turns out that MatAXPY fails but did not raise an exception and I did not check the error code returned from MatAXPY. In my code it give error code 60 in a MWE I actually get error code 56, with 
>  
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: No support for this operation for this object type
> [1]PETSC ERROR: Mat type elemental
>  
>  
> Prior to 646531bb84 MatAXPY worked with  Y = MATELEMENTAL and X= MATMPIAIJ  not raising a nonzero error code. 
>  
> I don't see any issue with the commit.
> Did you recompile petsc ? Can you send us a MWE to reproduce the error? Have you checked under valgrind?
>  
> Il Lun 25 Feb 2019, 06:58 Marius Buerkle via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> ha scritto:
> No excatly same source on my side. It works with commits prior to 646531bb84  and fails starting from 646531bb84.
> 
>  
>  
> On Mon, Feb 25, 2019 at 9:00 AM Marius Buerkle via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> Hi,
>  
> After commit 646531bb84 I get the following error message when calling MatLUFactorNumeric. It worked without a problem before.
>  
> [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [6]PETSC ERROR: Nonconforming object sizes
> [6]PETSC ERROR: Non conforming matrix add: local sizes 51 x 54, 51 x 54
>  
> This seems impossible since it reports above that the sizes are the same, but the error says they are different. Is it possible you source
> got accidentally changed?
>  
>   Matt
>  
> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
> [6]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1412-g646531bb84  GIT Date: 2019-02-07 21:47:41 +0300
> [6]PETSC ERROR: /home/marius/prog/ownstuff/fortran/programs/transomat_dev/transomat/transomat on a  named tono-hpc1 by marius Mon Feb 25 22:43:23 2019
> [6]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_opt_with_slepc --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpi
> cc --COPTFLAGS="-g -O0  -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0  -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0  -traceback " --with-mpi=1 --with-x=0 --download-p
> armetis=1 --download-metis=1 --download-superlu_dist=1 --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251 --download-hwloc=1 --download-sowing=
> 1 --with-openmp=0 --with-pthread=0 --download-elemental=1 --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff --with-cxx-dialect=c++11 --with-scala
> r-type=complex --with-debugging=1 --with-valgrind=0 --with-blaslapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scal
> apack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib=" -L/home/marius/intel/compilers
> _and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -
> ldl" --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.
> 1.144/linux/mkl --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_an
> d_libraries_2019.1.144/linux/mkl
> [6]PETSC ERROR: #1 MatAXPY() line 39 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
>  
> best,
> Marus
>  
>  
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>  
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190225/88fbb724/attachment-0001.html>

From mbuerkle at web.de  Tue Feb 26 00:50:43 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Tue, 26 Feb 2019 07:50:43 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
Message-ID: <trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190226/a6723d60/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error_output.tar.gz
Type: application/octet-stream
Size: 30374 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190226/a6723d60/attachment-0001.obj>

From eda.oktay at metu.edu.tr  Tue Feb 26 01:49:43 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Tue, 26 Feb 2019 10:49:43 +0300
Subject: [petsc-users] Finding nonzero indices
In-Reply-To: <CAMYG4Gk55rD9yG2o_=03N=485rGSsQMXpU=JsBJO3O8MByOxfQ@mail.gmail.com>
References: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
	<CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>
	<CA+vu_XiFCJ+FZFLqzXDs5G7g6hXXfOyjNSLXOLaqm1F6Zk1u8Q@mail.gmail.com>
	<CAMYG4Gk55rD9yG2o_=03N=485rGSsQMXpU=JsBJO3O8MByOxfQ@mail.gmail.com>
Message-ID: <CA+vu_XhAYY5LCyPVABaROL=ebowRbAQmkM7J8vZf9hm3EdThcQ@mail.gmail.com>

Okay, but here is another silly question:

>From MatGetRow(), I get number of nonzeros in row, column indices and
values right? Also, PetscBinaryWrite() needs a buffer p and the number of
items to write n.

I need both row,column indices and values. So, what should I write into
PetscBinaryWrite? Does this for loop give me these 3 arrays (arrays consist
of row indices,column indices and values)?

Eda

Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 18:37 tarihinde ?unu
yazd?:

> On Mon, Feb 25, 2019 at 10:20 AM Eda Oktay <eda.oktay at metu.edu.tr> wrote:
>
>> Thank you for your answer Matt, but I have some questions about
>>> PetscBinaryWrite() since I am a new user.
>>>
>>> Since I am going to use this function, I guess I have to open a new
>>> binary file with PetscViewerBinaryOpen
>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html#PetscViewerBinaryOpen>
>>>  and  PetscViewerBinaryGetDescriptor
>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryGetDescriptor.html#PetscViewerBinaryGetDescriptor>,
>>> and destroy viewer by  PetscViewerDestroy
>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerDestroy.html#PetscViewerDestroy>.
>>> Do I write these functions into for loop or not?
>>>
>>
> No, they go outside.
>
>   Thanks,
>
>     Matt
>
>
>> Eda
>>>
>>> Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 17:31 tarihinde
>>> ?unu yazd?:
>>>
>>>> On Mon, Feb 25, 2019 at 5:25 AM Eda Oktay via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am trying to find the indices (both row and column separately) of
>>>>> nonzero entries of a sparse matrix in Petsc Binary Format. I found
>>>>> MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
>>>>> this is what I want or not.
>>>>>
>>>>
>>>> I would not do it that way, since it is fragile. I would write a loop
>>>> like this:
>>>>
>>>> MAtGetOwnershipRange();
>>>> for (row = rStart; row < rEnd; ++row) {
>>>>   MatGetRow()
>>>>   PetscBinaryWrite()
>>>>   MatRestoreRow()
>>>> }
>>>>
>>>> This will work for any matrix.
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Best regards,
>>>>>
>>>>> Eda
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190226/69d4ee8b/attachment.html>

From dave.mayhem23 at gmail.com  Tue Feb 26 02:01:50 2019
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 26 Feb 2019 09:01:50 +0100
Subject: [petsc-users] Finding nonzero indices
In-Reply-To: <CA+vu_XhAYY5LCyPVABaROL=ebowRbAQmkM7J8vZf9hm3EdThcQ@mail.gmail.com>
References: <CA+vu_Xi69AUiVOELK1HC3fffssdT7dgsJv_hgm6VNJNc2p5gKg@mail.gmail.com>
	<CAMYG4Gk46=qGzTWSX-RjuhQkOKdtnksqBbFOB_voQBY_6ZyTqA@mail.gmail.com>
	<CA+vu_XiFCJ+FZFLqzXDs5G7g6hXXfOyjNSLXOLaqm1F6Zk1u8Q@mail.gmail.com>
	<CAMYG4Gk55rD9yG2o_=03N=485rGSsQMXpU=JsBJO3O8MByOxfQ@mail.gmail.com>
	<CA+vu_XhAYY5LCyPVABaROL=ebowRbAQmkM7J8vZf9hm3EdThcQ@mail.gmail.com>
Message-ID: <CAJ98EDq7Y5hC3GgmAs76zbRYu-6WSyUbCT3BjmonRWWOAgPSNA@mail.gmail.com>

On Tue, 26 Feb 2019 at 08:50, Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Okay, but here is another silly question:
>
> From MatGetRow(), I get number of nonzeros in row, column indices and
> values right?
>

Yes

Also, PetscBinaryWrite() needs a buffer p and the number of items to write
> n.
>
> I need both row,column indices and values.
>

Row index was proved to MatGetRow as an input arg - MatGetRow returns the
rest of what you need.


So, what should I write into PetscBinaryWrite?
>

That's up to you. You are in control of the format of your binary file.


Does this for loop give me these 3 arrays (arrays consist of row
> indices,column indices and values)?
>

Yes but you don't have an array for row indices - you only have a single
value. If you use Matt's loop you would be writing the matrix row by row to
your binary file.



> Eda
>
> Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 18:37 tarihinde
> ?unu yazd?:
>
>> On Mon, Feb 25, 2019 at 10:20 AM Eda Oktay <eda.oktay at metu.edu.tr> wrote:
>>
>>> Thank you for your answer Matt, but I have some questions about
>>>> PetscBinaryWrite() since I am a new user.
>>>>
>>>> Since I am going to use this function, I guess I have to open a new
>>>> binary file with PetscViewerBinaryOpen
>>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryOpen.html#PetscViewerBinaryOpen>
>>>>  and  PetscViewerBinaryGetDescriptor
>>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerBinaryGetDescriptor.html#PetscViewerBinaryGetDescriptor>,
>>>> and destroy viewer by  PetscViewerDestroy
>>>> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerDestroy.html#PetscViewerDestroy>.
>>>> Do I write these functions into for loop or not?
>>>>
>>>
>> No, they go outside.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Eda
>>>>
>>>> Matthew Knepley <knepley at gmail.com>, 25 ?ub 2019 Pzt, 17:31 tarihinde
>>>> ?unu yazd?:
>>>>
>>>>> On Mon, Feb 25, 2019 at 5:25 AM Eda Oktay via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am trying to find the indices (both row and column separately) of
>>>>>> nonzero entries of a sparse matrix in Petsc Binary Format. I found
>>>>>> MatGetSeqNonzeroStructure but it gives me a struct and I don't know whether
>>>>>> this is what I want or not.
>>>>>>
>>>>>
>>>>> I would not do it that way, since it is fragile. I would write a loop
>>>>> like this:
>>>>>
>>>>> MAtGetOwnershipRange();
>>>>> for (row = rStart; row < rEnd; ++row) {
>>>>>   MatGetRow()
>>>>>   PetscBinaryWrite()
>>>>>   MatRestoreRow()
>>>>> }
>>>>>
>>>>> This will work for any matrix.
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Eda
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190226/03a6946a/attachment.html>

From yyang85 at stanford.edu  Wed Feb 27 01:06:19 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Wed, 27 Feb 2019 07:06:19 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address which
 was not malloc()-ed
Message-ID: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>

Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don't really know what's wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/fa9da29f/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: domain.cpp
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/fa9da29f/attachment-0001.ksh>

From knepley at gmail.com  Wed Feb 27 09:24:26 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 Feb 2019 10:24:26 -0500
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom
function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello team,
>
>
>
> I ran into the address sanitizer error that I hope you could help me with.
> I don?t really know what?s wrong with the way the code frees memory. The
> relevant code file is attached. The line number following domain.cpp
> specifically referenced to the vector _q, which seems a little odd, since
> some other vectors are constructed and freed the same way.
>
>
>
> ==1719==ERROR: AddressSanitizer: attempting free on address which was not
> malloc()-ed: 0x61f0000076c0 in thread T0
>
>     #0 0x7fbf195282ca in __interceptor_free
> (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
>
>     #1 0x7fbf1706f895 in PetscFreeAlign
> /home/yyy910805/petsc/src/sys/memory/mal.c:87
>
>     #2 0x7fbf1731a898 in VecDestroy_Seq
> /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
>
>     #3 0x7fbf1735f795 in VecDestroy
> /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
>
>     #4 0x40dd0a in Domain::~Domain()
> /home/yyy910805/scycle/source/domain.cpp:132
>
>     #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
>
>     #6 0x7fbf14d2082f in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
>
>     #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)
>
>
>
> 0x61f0000076c0 is located 1600 bytes inside of 3220-byte region
> [0x61f000007080,0x61f000007d14)
>
> allocated by thread T0 here:
>
>     #0 0x7fbf19528b32 in __interceptor_memalign
> (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
>
>     #1 0x7fbf1706f7e0 in PetscMallocAlign
> /home/yyy910805/petsc/src/sys/memory/mal.c:41
>
>     #2 0x7fbf17073022 in PetscTrMallocDefault
> /home/yyy910805/petsc/src/sys/memory/mtr.c:183
>
>     #3 0x7fbf170710a1 in PetscMallocA
> /home/yyy910805/petsc/src/sys/memory/mal.c:397
>
>     #4 0x7fbf17326fb0 in VecCreate_Seq
> /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
>
>     #5 0x7fbf1736f560 in VecSetType
> /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
>
>     #6 0x7fbf1731afae in VecDuplicate_Seq
> /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
>
>     #7 0x7fbf1735eff7 in VecDuplicate
> /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
>
>     #8 0x4130de in Domain::setFields()
> /home/yyy910805/scycle/source/domain.cpp:431
>
>     #9 0x40c60a in Domain::Domain(char const*)
> /home/yyy910805/scycle/source/domain.cpp:57
>
>     #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
>
>     #11 0x7fbf14d2082f in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
>
>
>
> SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
>
> ==1719==ABORTING
>
>
>
> Thanks very much!
>
> Yuyun
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/5ace6240/attachment.html>

From yyang85 at stanford.edu  Wed Feb 27 10:34:10 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Wed, 27 Feb 2019 16:34:10 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
Message-ID: <DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>

I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
How do I implement CHECK ALL RETURN CODES?

From: Matthew Knepley <knepley at gmail.com>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/d93f3af3/attachment-0001.html>

From stefano.zampini at gmail.com  Wed Feb 27 11:00:50 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 27 Feb 2019 09:00:50 -0800
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
Message-ID: <99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>

If you check ALL ERROR CODES by doing
ierr = _AnyPetscCall_();CHKERRQ(ierr);
The code will stop at the first error 

I see you are using openmp. Probably this is causing the problem, I don?t know what to say without a MWE.


> On Feb 25, 2019, at 10:50 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> 
> <error_output.tar.gz>


From fdkong.jd at gmail.com  Wed Feb 27 11:55:32 2019
From: fdkong.jd at gmail.com (Fande Kong)
Date: Wed, 27 Feb 2019 10:55:32 -0700
Subject: [petsc-users] Fwd: PETSC installation issues
In-Reply-To: <09180386-e88b-4776-94cc-1f0cc00e00f6@googlegroups.com>
References: <09180386-e88b-4776-94cc-1f0cc00e00f6@googlegroups.com>
Message-ID: <CAN5Wd-KAP+Bz59CmEavGvz-XtNq5nCErPSVR03TToCL_95FvrA@mail.gmail.com>

Hi Satish,

Do you have any idea why the configure failed?

Thanks,

Fande

On Mon, Feb 25, 2019 at 2:57 PM <ryan.c.cooper at uconn.edu> wrote:

> Greetings,
> I am having trouble installing Petsc with the given configuration options
> in the scripts/update_and_rebuild_petsc.sh. I have attached the
> confiugure.log. I am running Arch linux - kernel 4.20.12, mpi: gcc 8.2.1,
> mpif77/90: gnu fortran gcc 8.2.1.
>
> The main problem is that I keep getting errors for superlu_dist and
> scalapack that the downloaded file cannot be installed with the given
> configuration. Are there some other flags I can specify to compile and link
> the packages with Petsc?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "moose-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to moose-users+unsubscribe at googlegroups.com.
> Visit this group at https://groups.google.com/group/moose-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com
> <https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/36b55705/attachment-0001.html>
-------------- next part --------------
                              Pushing language C
                              Popping language C
                              Pushing language CUDA
                              Popping language CUDA
                              Pushing language Cxx
                              Popping language Cxx
                              Pushing language FC
                              Popping language FC

================================================================================
================================================================================
Starting configure run at Sat, 23 Feb 2019 15:55:51 -0500
Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 --CFLAGS="-fPIC -fopenmp" --CXXFLAGS="-fPIC -fopenmp" --FFLAGS="-fPIC -fopenmp" --FCFLAGS="-fPIC -fopenmp" --F90FLAGS="-fPIC -fopenmp" --F77FLAGS="-fPIC -fopenmp" PETSC_DIR=/home/ryan/projects/moose/petsc
Working directory: /home/ryan/projects/moose/petsc
Machine platform:
('Linux', 'fermi', '4.20.10-arch1-1-ARCH', '#1 SMP PREEMPT Fri Feb 15 17:49:06 UTC 2019', 'x86_64', '')
Python version:
2.7.15 (default, Jan 10 2019, 23:20:52) 
[GCC 8.2.1 20181127]
================================================================================
          Pushing language C
          Popping language C
          Pushing language CUDA
          Popping language CUDA
          Pushing language Cxx
          Popping language Cxx
          Pushing language FC
          Popping language FC
================================================================================
TEST configureExternalPackagesDir from config.framework(/home/ryan/projects/moose/petsc/config/BuildSystem/config/framework.py:836)
TESTING: configureExternalPackagesDir from config.framework(config/BuildSystem/config/framework.py:836)
================================================================================
TEST configureDebuggers from config.utilities.debuggers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/debuggers.py:22)
TESTING: configureDebuggers from config.utilities.debuggers(config/BuildSystem/config/utilities/debuggers.py:22)
  Find a default debugger and determine its arguments
Checking for program /home/ryan/bin/gdb...not found
Checking for program /usr/local/bin/gdb...not found
Checking for program /home/ryan/bin/gdb...not found
Checking for program /usr/local/bin/gdb...not found
Checking for program /usr/local/sbin/gdb...not found
Checking for program /usr/bin/gdb...found
                Defined make macro "GDB" to "/usr/bin/gdb"
Checking for program /home/ryan/bin/dbx...not found
Checking for program /usr/local/bin/dbx...not found
Checking for program /home/ryan/bin/dbx...not found
Checking for program /usr/local/bin/dbx...not found
Checking for program /usr/local/sbin/dbx...not found
Checking for program /usr/bin/dbx...not found
Checking for program /usr/bin/site_perl/dbx...not found
Checking for program /usr/bin/vendor_perl/dbx...not found
Checking for program /usr/bin/core_perl/dbx...not found
Checking for program /home/ryan/.local/bin/dbx...not found
Checking for program /home/ryan/bin/xdb...not found
Checking for program /usr/local/bin/xdb...not found
Checking for program /home/ryan/bin/xdb...not found
Checking for program /usr/local/bin/xdb...not found
Checking for program /usr/local/sbin/xdb...not found
Checking for program /usr/bin/xdb...not found
Checking for program /usr/bin/site_perl/xdb...not found
Checking for program /usr/bin/vendor_perl/xdb...not found
Checking for program /usr/bin/core_perl/xdb...not found
Checking for program /home/ryan/.local/bin/xdb...not found
Executing: uname -s
stdout: Linux
              Defined make macro "DSYMUTIL" to "true"
              Defined "USE_GDB_DEBUGGER" to "1"
================================================================================
TEST configureGit from config.sourceControl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/sourceControl.py:24)
TESTING: configureGit from config.sourceControl(config/BuildSystem/config/sourceControl.py:24)
  Find the Git executable
Checking for program /home/ryan/bin/git...not found
Checking for program /usr/local/bin/git...not found
Checking for program /home/ryan/bin/git...not found
Checking for program /usr/local/bin/git...not found
Checking for program /usr/local/sbin/git...not found
Checking for program /usr/bin/git...found
                Defined make macro "GIT" to "git"
Executing: git --version
stdout: git version 2.20.1
================================================================================
TEST configureMercurial from config.sourceControl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/sourceControl.py:35)
TESTING: configureMercurial from config.sourceControl(config/BuildSystem/config/sourceControl.py:35)
  Find the Mercurial executable
Checking for program /home/ryan/bin/hg...not found
Checking for program /usr/local/bin/hg...not found
Checking for program /home/ryan/bin/hg...not found
Checking for program /usr/local/bin/hg...not found
Checking for program /usr/local/sbin/hg...not found
Checking for program /usr/bin/hg...found
                Defined make macro "HG" to "hg"
Executing: hg version -q
stdout: Mercurial Distributed SCM (version 4.9)
================================================================================
TEST configureCLanguage from PETSc.options.languages(/home/ryan/projects/moose/petsc/config/PETSc/options/languages.py:27)
TESTING: configureCLanguage from PETSc.options.languages(config/PETSc/options/languages.py:27)
  Choose whether to compile the PETSc library using a C or C++ compiler
            C language is C
              Defined "CLANGUAGE_C" to "1"
================================================================================
TEST configureDirectories from PETSc.options.petscdir(/home/ryan/projects/moose/petsc/config/PETSc/options/petscdir.py:23)
TESTING: configureDirectories from PETSc.options.petscdir(config/PETSc/options/petscdir.py:23)
  Checks PETSC_DIR and sets if not set
            Version Information:
            #define PETSC_VERSION_RELEASE    1
            #define PETSC_VERSION_MAJOR      3
            #define PETSC_VERSION_MINOR      10
            #define PETSC_VERSION_SUBMINOR   2
            #define PETSC_VERSION_PATCH      0
            #define PETSC_VERSION_DATE       "unknown"
            #define PETSC_VERSION_GIT        "unknown"
            #define PETSC_VERSION_DATE_GIT   "unknown"
            #define PETSC_VERSION_EQ(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_ PETSC_VERSION_EQ
            #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR)          \
            #define PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_GE(MAJOR,MINOR,SUBMINOR) \
================================================================================
TEST getDatafilespath from PETSc.options.dataFilesPath(/home/ryan/projects/moose/petsc/config/PETSc/options/dataFilesPath.py:29)
TESTING: getDatafilespath from PETSc.options.dataFilesPath(config/PETSc/options/dataFilesPath.py:29)
  Checks what DATAFILESPATH should be
================================================================================
TEST configureInstallationMethod from PETSc.options.petscclone(/home/ryan/projects/moose/petsc/config/PETSc/options/petscclone.py:20)
TESTING: configureInstallationMethod from PETSc.options.petscclone(config/PETSc/options/petscclone.py:20)
            lib/petsc/bin/maint exists. This appears to be a repository clone
            .git directory exists
Executing: cd /home/ryan/projects/moose/petsc && git describe --match "v*"
stdout: v3.10.2-76-ge1487ad806
Executing: cd /home/ryan/projects/moose/petsc && git log -1 --pretty=format:%H
stdout: e1487ad8065c1157da7a4b7f82e219b7fa0de74f
Executing: cd /home/ryan/projects/moose/petsc && git log -1 --pretty=format:%ci
stdout: 2018-12-02 07:19:50 +0100
Executing: cd /home/ryan/projects/moose/petsc && git branch
stdout:
* (HEAD detached at e1487ad806)
  master
              Defined "VERSION_GIT" to ""v3.10.2-76-ge1487ad806""
              Defined "VERSION_DATE_GIT" to ""2018-12-02 07:19:50 +0100""
              Defined "VERSION_BRANCH_GIT" to ""(HEAD detached at e1487ad806)""
================================================================================
TEST configureArchitecture from PETSc.options.arch(/home/ryan/projects/moose/petsc/config/PETSc/options/arch.py:36)
TESTING: configureArchitecture from PETSc.options.arch(config/PETSc/options/arch.py:36)
  Checks PETSC_ARCH and sets if not set
================================================================================
TEST setInstallDir from PETSc.options.installDir(/home/ryan/projects/moose/petsc/config/PETSc/options/installDir.py:36)
TESTING: setInstallDir from PETSc.options.installDir(config/PETSc/options/installDir.py:36)
   setup installDir to either prefix or if that is not set to PETSC_DIR/PETSC_ARCH
              Defined make macro "PREFIXDIR" to "/home/ryan/projects/moose/petsc/linux-c-opt"
================================================================================
TEST saveReconfigure from PETSc.options.installDir(/home/ryan/projects/moose/petsc/config/PETSc/options/installDir.py:80)
TESTING: saveReconfigure from PETSc.options.installDir(config/PETSc/options/installDir.py:80)
================================================================================
TEST cleanConfDir from PETSc.options.installDir(/home/ryan/projects/moose/petsc/config/PETSc/options/installDir.py:73)
TESTING: cleanConfDir from PETSc.options.installDir(config/PETSc/options/installDir.py:73)
================================================================================
TEST configureInstallDir from PETSc.options.installDir(/home/ryan/projects/moose/petsc/config/PETSc/options/installDir.py:57)
TESTING: configureInstallDir from PETSc.options.installDir(config/PETSc/options/installDir.py:57)
  Makes  installDir subdirectories if it does not exist for both prefix install location and PETSc work install location
            Changed persistence directory to /home/ryan/projects/moose/petsc/linux-c-opt/lib/petsc/conf
================================================================================
TEST restoreReconfigure from PETSc.options.installDir(/home/ryan/projects/moose/petsc/config/PETSc/options/installDir.py:93)
TESTING: restoreReconfigure from PETSc.options.installDir(config/PETSc/options/installDir.py:93)
================================================================================
TEST setExternalPackagesDir from PETSc.options.externalpackagesdir(/home/ryan/projects/moose/petsc/config/PETSc/options/externalpackagesdir.py:15)
TESTING: setExternalPackagesDir from PETSc.options.externalpackagesdir(config/PETSc/options/externalpackagesdir.py:15)
================================================================================
TEST cleanExternalpackagesDir from PETSc.options.externalpackagesdir(/home/ryan/projects/moose/petsc/config/PETSc/options/externalpackagesdir.py:22)
TESTING: cleanExternalpackagesDir from PETSc.options.externalpackagesdir(config/PETSc/options/externalpackagesdir.py:22)
================================================================================
TEST printEnvVariables from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1638)
TESTING: printEnvVariables from config.setCompilers(config/BuildSystem/config/setCompilers.py:1638)
            **** printenv ****
CSF_SHMessage=/usr/share/opencascade/resources/SHMessage
DRAWHOME=/usr/share/opencascade/resources/DrawResources
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/ryan
KDE_SESSION_VERSION=5
CSF_StandardLiteDefaults=/usr/share/opencascade/resources/StdResource
MMGT_CLEAR=1
MSC_LICENSE_FILE=27500 at engr-license3.engr.uconn.edu
XCURSOR_SIZE=0
CSF_OCCTResourcePath=/usr/share/opencascade/resources
PWD=/home/ryan/projects/moose/petsc
LOGNAME=ryan
USER=ryan
CSF_XCAFDefaults=/usr/share/opencascade/resources/StdResource
XDG_VTNR=7
CSF_XmlOcafResource=/usr/share/opencascade/resources/XmlOcafResource
DISPLAY=:0
KDE_FULL_SESSION=true
LANG=en_US.UTF-8
PROFILEHOME=
TERM=screen-256color
SHELL=/usr/bin/fish
COLORFGBG=15;0
XAUTHORITY=/home/ryan/.Xauthority
LANGUAGE=en_US
SESSION_MANAGER=local/fermi:@/tmp/.ICE-unix/750,unix/fermi:/tmp/.ICE-unix/750
SHLVL=3
XDG_SESSION_ID=2
XDG_SESSION_TYPE=x11
CSF_IGESDefaults=/usr/share/opencascade/resources/XSTEPResource
SHELL_SESSION_ID=c9813393d6f74f58ba6ea0cb0b929cc1
HG=/usr/bin/hg
KONSOLE_DBUS_WINDOW=/Windows/1
XDG_RUNTIME_DIR=/run/user/1000
PETSC_DIR=/opt/petsc/linux-c-opt
CASROOT=/usr
EDITOR=vim
CSF_PluginDefaults=/usr/share/opencascade/resources/StdResource
XCURSOR_THEME=breeze_cursors
KONSOLE_PROFILE_NAME=Profile 2
HOME=/home/ryan
KONSOLE_DBUS_SESSION=/Sessions/1
QT_AUTO_SCREEN_SCALE_FACTOR=0
PATH=/home/ryan/bin:/usr/local/bin:/home/ryan/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/ryan/.local/bin
XDG_SESSION_DESKTOP=plasma
KDE_SESSION_UID=1000
CSF_XSMessage=/usr/share/opencascade/resources/XSMessage
CSF_TObjDefaults=/usr/share/opencascade/resources/StdResource
CSF_STEPDefaults=/usr/share/opencascade/resources/XSTEPResource
KONSOLE_DBUS_SERVICE=:1.67
DESKTOP_SESSION=plasma
TMUX=/tmp//tmux-1000/default,1253,0
GDMSESSION=plasma
PETSC_ARCH=linux-c-opt
VISUAL=vim
CSF_MIGRATION_TYPES=/usr/share/opencascade/resources/StdResource/MigrationSheet.txt
CSF_LANGUAGE=us
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
CSF_TObjMessage=/usr/share/opencascade/resources/TObj
XDG_CURRENT_DESKTOP=KDE
GS_LIB=/home/ryan/.fonts
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins
CSF_ShadersDirectory=/usr/share/opencascade/resources/Shaders
CSF_MDTVTexturesDirectory=/usr/share/opencascade/resources/Textures
WINDOWID=4194311
GTK_MODULES=canberra-gtk-module
CSF_EXCEPTION_PROMPT=1
DESKTOP_AUTOSTART_ID=
GDM_LANG=en_US.UTF-8
DRAWDEFAULT=/usr/share/opencascade/resources/DrawResources/DrawDefault
KONSOLE_VERSION=181202
CSF_DrawPluginDefaults=/usr/share/opencascade/resources/DrawResources
XDG_DATA_DIRS=/usr/share:/usr/share:/usr/local/share
CSF_StandardDefaults=/usr/share/opencascade/resources/StdResource
XDG_SESSION_CLASS=user
COLORTERM=truecolor
MAIL=/var/spool/mail/ryan
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TMUX_PANE=%8
XDG_SEAT=seat0
================================================================================
TEST resetEnvCompilers from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1645)
TESTING: resetEnvCompilers from config.setCompilers(config/BuildSystem/config/setCompilers.py:1645)
================================================================================
TEST checkEnvCompilers from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1675)
TESTING: checkEnvCompilers from config.setCompilers(config/BuildSystem/config/setCompilers.py:1675)
================================================================================
TEST checkMPICompilerOverride from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1608)
TESTING: checkMPICompilerOverride from config.setCompilers(config/BuildSystem/config/setCompilers.py:1608)
  Check if --with-mpi-dir is used along with CC CXX or FC compiler options.
    This usually prevents mpi compilers from being used - so issue a warning
================================================================================
TEST requireMpiLdPath from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1629)
TESTING: requireMpiLdPath from config.setCompilers(config/BuildSystem/config/setCompilers.py:1629)
  OpenMPI wrappers require LD_LIBRARY_PATH set
================================================================================
TEST checkVendor from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:444)
TESTING: checkVendor from config.setCompilers(config/BuildSystem/config/setCompilers.py:444)
  Determine the compiler vendor
            Compiler vendor is ""
================================================================================
TEST checkInitialFlags from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:454)
TESTING: checkInitialFlags from config.setCompilers(config/BuildSystem/config/setCompilers.py:454)
  Initialize the compiler and linker flags
              Pushing language C
            Initialized CFLAGS to -fPIC -fopenmp
            Initialized CFLAGS to -fPIC -fopenmp
            Initialized LDFLAGS to 
              Popping language C
              Pushing language CUDA
            Initialized CUDAFLAGS to 
            Initialized CUDAFLAGS to 
            Initialized LDFLAGS to 
              Popping language CUDA
              Pushing language Cxx
            Initialized CXXFLAGS to -fPIC -fopenmp
            Initialized CXX_CXXFLAGS to 
            Initialized LDFLAGS to 
              Popping language Cxx
              Pushing language FC
            Initialized FFLAGS to -fPIC -fopenmp
            Initialized FFLAGS to -fPIC -fopenmp
            Initialized LDFLAGS to 
              Popping language FC
            Initialized CPPFLAGS to 
            Initialized CUDAPPFLAGS to -Wno-deprecated-gpu-targets
            Initialized CXXCPPFLAGS to 
            Initialized CC_LINKER_FLAGS to []
            Initialized CXX_LINKER_FLAGS to []
            Initialized FC_LINKER_FLAGS to []
            Initialized CUDAC_LINKER_FLAGS to []
            Initialized sharedLibraryFlags to []
            Initialized dynamicLibraryFlags to []
================================================================================
TEST checkCCompiler from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:587)
TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:587)
  Locate a functional C compiler
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /usr/local/sbin/mpicc...not found
Checking for program /usr/bin/mpicc...found
                Defined make macro "CC" to "mpicc"
                Pushing language C
                            All intermediate test results are stored in /tmp/petsc-VuReVk
                          All intermediate test results are stored in /tmp/petsc-VuReVk/config.setCompilers
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-VuReVk/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
                Popping language C
================================================================================
TEST checkCPreprocessor from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:620)
TESTING: checkCPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:620)
  Locate a functional C preprocessor
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /usr/local/sbin/mpicc...not found
Checking for program /usr/bin/mpicc...found
                Defined make macro "CPP" to "mpicc -E"
              Pushing language C
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.setCompilers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.setCompilers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.setCompilers/conftest.c" 2
# 1 "/usr/include/stdlib.h" 1 3 4
# 25 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 26 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 32 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 40 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 41 "/usr/include/stdlib.h" 2 3 4
# 55 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 56 "/usr/include/stdlib.h" 2 3 4
typedef struct
  {
    int quot;
    int rem;
  } div_t;
typedef struct
  {
    long int quot;
    long int rem;
  } ldiv_t;
__extension__ typedef struct
  {
    long long int quot;
    long long int rem;
  } lldiv_t;
# 97 "/usr/include/stdlib.h" 3 4
extern size_t __ctype_get_mb_cur_max (void) __attribute__ ((__nothrow__ , __leaf__)) ;
extern double atof (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern int atoi (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern long int atol (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
__extension__ extern long long int atoll (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern double strtod (const char *__restrict __nptr,
        char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern float strtof (const char *__restrict __nptr,
       char **__restrict __endptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long double strtold (const char *__restrict __nptr,
       char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 176 "/usr/include/stdlib.h" 3 4
extern long int strtol (const char *__restrict __nptr,
   char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern unsigned long int strtoul (const char *__restrict __nptr,
      char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoq (const char *__restrict __nptr,
        char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtouq (const char *__restrict __nptr,
           char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoll (const char *__restrict __nptr,
         char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtoull (const char *__restrict __nptr,
     char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 385 "/usr/include/stdlib.h" 3 4
extern char *l64a (long int __n) __attribute__ ((__nothrow__ , __leaf__)) ;
extern long int a64l (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 395 "/usr/include/stdlib.h" 2 3 4
extern long int random (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srandom (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern char *initstate (unsigned int __seed, char *__statebuf,
   size_t __statelen) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *setstate (char *__statebuf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct random_data
  {
    int32_t *fptr;
    int32_t *rptr;
    int32_t *state;
    int rand_type;
    int rand_deg;
    int rand_sep;
    int32_t *end_ptr;
  };
extern int random_r (struct random_data *__restrict __buf,
       int32_t *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srandom_r (unsigned int __seed, struct random_data *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int initstate_r (unsigned int __seed, char *__restrict __statebuf,
   size_t __statelen,
   struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern int setstate_r (char *__restrict __statebuf,
         struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int rand (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srand (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern int rand_r (unsigned int *__seed) __attribute__ ((__nothrow__ , __leaf__));
extern double drand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern double erand48 (unsigned short int __xsubi[3]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int lrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int nrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int mrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int jrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void srand48 (long int __seedval) __attribute__ ((__nothrow__ , __leaf__));
extern unsigned short int *seed48 (unsigned short int __seed16v[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void lcong48 (unsigned short int __param[7]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct drand48_data
  {
    unsigned short int __x[3];
    unsigned short int __old_x[3];
    unsigned short int __c;
    unsigned short int __init;
    __extension__ unsigned long long int __a;
  };
extern int drand48_r (struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int erand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int nrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int mrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int jrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srand48_r (long int __seedval, struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int seed48_r (unsigned short int __seed16v[3],
       struct drand48_data *__buffer) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lcong48_r (unsigned short int __param[7],
        struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *malloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
# 563 "/usr/include/stdlib.h" 3 4
extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/alloca.h" 1 3 4
# 24 "/usr/include/alloca.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 25 "/usr/include/alloca.h" 2 3 4
extern void *alloca (size_t __size) __attribute__ ((__nothrow__ , __leaf__));
# 567 "/usr/include/stdlib.h" 2 3 4
extern void *valloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern void *aligned_alloc (size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__alloc_size__ (2))) ;
extern void abort (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern int atexit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int at_quick_exit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int on_exit (void (*__func) (int __status, void *__arg), void *__arg)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void quick_exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void _Exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern char *getenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 644 "/usr/include/stdlib.h" 3 4
extern int putenv (char *__string) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int setenv (const char *__name, const char *__value, int __replace)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int unsetenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int clearenv (void) __attribute__ ((__nothrow__ , __leaf__));
# 672 "/usr/include/stdlib.h" 3 4
extern char *mktemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 685 "/usr/include/stdlib.h" 3 4
extern int mkstemp (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 707 "/usr/include/stdlib.h" 3 4
extern int mkstemps (char *__template, int __suffixlen) __attribute__ ((__nonnull__ (1))) ;
# 728 "/usr/include/stdlib.h" 3 4
extern char *mkdtemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 781 "/usr/include/stdlib.h" 3 4
extern int system (const char *__command) ;
# 797 "/usr/include/stdlib.h" 3 4
extern char *realpath (const char *__restrict __name,
         char *__restrict __resolved) __attribute__ ((__nothrow__ , __leaf__)) ;
typedef int (*__compar_fn_t) (const void *, const void *);
# 817 "/usr/include/stdlib.h" 3 4
extern void *bsearch (const void *__key, const void *__base,
        size_t __nmemb, size_t __size, __compar_fn_t __compar)
     __attribute__ ((__nonnull__ (1, 2, 5))) ;
extern void qsort (void *__base, size_t __nmemb, size_t __size,
     __compar_fn_t __compar) __attribute__ ((__nonnull__ (1, 4)));
# 837 "/usr/include/stdlib.h" 3 4
extern int abs (int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern long int labs (long int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern long long int llabs (long long int __x)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern div_t div (int __numer, int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern ldiv_t ldiv (long int __numer, long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern lldiv_t lldiv (long long int __numer,
        long long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
# 869 "/usr/include/stdlib.h" 3 4
extern char *ecvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *fcvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *gcvt (double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern char *qecvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qfcvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qgcvt (long double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern int ecvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int fcvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qecvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qfcvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int mblen (const char *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int mbtowc (wchar_t *__restrict __pwc,
     const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int wctomb (char *__s, wchar_t __wchar) __attribute__ ((__nothrow__ , __leaf__));
extern size_t mbstowcs (wchar_t *__restrict __pwcs,
   const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern size_t wcstombs (char *__restrict __s,
   const wchar_t *__restrict __pwcs, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__));
extern int rpmatch (const char *__response) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 954 "/usr/include/stdlib.h" 3 4
extern int getsubopt (char **__restrict __optionp,
        char *const *__restrict __tokens,
        char **__restrict __valuep)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3))) ;
# 1000 "/usr/include/stdlib.h" 3 4
extern int getloadavg (double __loadavg[], int __nelem)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1010 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/stdlib-float.h" 1 3 4
# 1011 "/usr/include/stdlib.h" 2 3 4
# 1020 "/usr/include/stdlib.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.setCompilers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
              Popping language C
================================================================================
TEST checkCUDACompiler from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:661)
TESTING: checkCUDACompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:661)
  Locate a functional CUDA compiler
================================================================================
TEST checkCUDAPreprocessor from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:701)
TESTING: checkCUDAPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:701)
  Locate a functional CUDA preprocessor
================================================================================
TEST checkCxxCompiler from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:814)
TESTING: checkCxxCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:814)
  Locate a functional Cxx compiler
Checking for program /home/ryan/bin/mpicxx...not found
Checking for program /usr/local/bin/mpicxx...not found
Checking for program /home/ryan/bin/mpicxx...not found
Checking for program /usr/local/bin/mpicxx...not found
Checking for program /usr/local/sbin/mpicxx...not found
Checking for program /usr/bin/mpicxx...found
                Defined make macro "CXX" to "mpicxx"
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                        Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                            Pushing language CXX
                            Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-VuReVk/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
                Popping language Cxx
================================================================================
TEST checkCxxPreprocessor from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:852)
TESTING: checkCxxPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:852)
  Locate a functional Cxx preprocessor
Checking for program /home/ryan/bin/mpicxx...not found
Checking for program /usr/local/bin/mpicxx...not found
Checking for program /home/ryan/bin/mpicxx...not found
Checking for program /usr/local/bin/mpicxx...not found
Checking for program /usr/local/sbin/mpicxx...not found
Checking for program /usr/bin/mpicxx...found
                Defined make macro "CXXCPP" to "mpicxx -E"
              Pushing language Cxx
Executing: mpicxx -E  -I/tmp/petsc-VuReVk/config.setCompilers   /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
stdout:
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conftest.cc"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conftest.cc"
# 1 "/tmp/petsc-VuReVk/config.setCompilers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.setCompilers/conftest.cc" 2
# 1 "/tmp/petsc-VuReVk/config.setCompilers/conffix.h" 1
extern "C" {
}
# 3 "/tmp/petsc-VuReVk/config.setCompilers/conftest.cc" 2
# 1 "/usr/include/c++/8.2.1/cstdlib" 1 3
# 39 "/usr/include/c++/8.2.1/cstdlib" 3
       
# 40 "/usr/include/c++/8.2.1/cstdlib" 3
# 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 1 3
# 236 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3
# 236 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3
namespace std
{
  typedef long unsigned int size_t;
  typedef long int ptrdiff_t;
  typedef decltype(nullptr) nullptr_t;
}
# 258 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3
namespace std
{
  inline namespace __cxx11 __attribute__((__abi_tag__ ("cxx11"))) { }
}
namespace __gnu_cxx
{
  inline namespace __cxx11 __attribute__((__abi_tag__ ("cxx11"))) { }
}
# 508 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3
# 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 1 3
# 39 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 3
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 40 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 2 3
# 509 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 2 3
# 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/cpu_defines.h" 1 3
# 512 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 2 3
# 42 "/usr/include/c++/8.2.1/cstdlib" 2 3
# 75 "/usr/include/c++/8.2.1/cstdlib" 3
# 1 "/usr/include/stdlib.h" 1 3 4
# 25 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 26 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 32 "/usr/include/stdlib.h" 2 3 4
extern "C" {
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 40 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 41 "/usr/include/stdlib.h" 2 3 4
# 55 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 74 "/usr/include/bits/floatn.h" 3 4
typedef _Complex float __cfloat128 __attribute__ ((__mode__ (__TC__)));
# 86 "/usr/include/bits/floatn.h" 3 4
typedef __float128 _Float128;
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 214 "/usr/include/bits/floatn-common.h" 3 4
typedef float _Float32;
# 251 "/usr/include/bits/floatn-common.h" 3 4
typedef double _Float64;
# 268 "/usr/include/bits/floatn-common.h" 3 4
typedef double _Float32x;
# 285 "/usr/include/bits/floatn-common.h" 3 4
typedef long double _Float64x;
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 56 "/usr/include/stdlib.h" 2 3 4
typedef struct
  {
    int quot;
    int rem;
  } div_t;
typedef struct
  {
    long int quot;
    long int rem;
  } ldiv_t;
__extension__ typedef struct
  {
    long long int quot;
    long long int rem;
  } lldiv_t;
# 97 "/usr/include/stdlib.h" 3 4
extern size_t __ctype_get_mb_cur_max (void) throw () ;
extern double atof (const char *__nptr)
     throw () __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern int atoi (const char *__nptr)
     throw () __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern long int atol (const char *__nptr)
     throw () __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
__extension__ extern long long int atoll (const char *__nptr)
     throw () __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern double strtod (const char *__restrict __nptr,
        char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
extern float strtof (const char *__restrict __nptr,
       char **__restrict __endptr) throw () __attribute__ ((__nonnull__ (1)));
extern long double strtold (const char *__restrict __nptr,
       char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
# 140 "/usr/include/stdlib.h" 3 4
extern _Float32 strtof32 (const char *__restrict __nptr,
     char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
extern _Float64 strtof64 (const char *__restrict __nptr,
     char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
extern _Float128 strtof128 (const char *__restrict __nptr,
       char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
extern _Float32x strtof32x (const char *__restrict __nptr,
       char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
extern _Float64x strtof64x (const char *__restrict __nptr,
       char **__restrict __endptr)
     throw () __attribute__ ((__nonnull__ (1)));
# 176 "/usr/include/stdlib.h" 3 4
extern long int strtol (const char *__restrict __nptr,
   char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
extern unsigned long int strtoul (const char *__restrict __nptr,
      char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoq (const char *__restrict __nptr,
        char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtouq (const char *__restrict __nptr,
           char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoll (const char *__restrict __nptr,
         char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtoull (const char *__restrict __nptr,
     char **__restrict __endptr, int __base)
     throw () __attribute__ ((__nonnull__ (1)));
extern int strfromd (char *__dest, size_t __size, const char *__format,
       double __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfromf (char *__dest, size_t __size, const char *__format,
       float __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfroml (char *__dest, size_t __size, const char *__format,
       long double __f)
     throw () __attribute__ ((__nonnull__ (3)));
# 232 "/usr/include/stdlib.h" 3 4
extern int strfromf32 (char *__dest, size_t __size, const char * __format,
         _Float32 __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfromf64 (char *__dest, size_t __size, const char * __format,
         _Float64 __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfromf128 (char *__dest, size_t __size, const char * __format,
   _Float128 __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfromf32x (char *__dest, size_t __size, const char * __format,
   _Float32x __f)
     throw () __attribute__ ((__nonnull__ (3)));
extern int strfromf64x (char *__dest, size_t __size, const char * __format,
   _Float64x __f)
     throw () __attribute__ ((__nonnull__ (3)));
# 272 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 273 "/usr/include/stdlib.h" 2 3 4
extern long int strtol_l (const char *__restrict __nptr,
     char **__restrict __endptr, int __base,
     locale_t __loc) throw () __attribute__ ((__nonnull__ (1, 4)));
extern unsigned long int strtoul_l (const char *__restrict __nptr,
        char **__restrict __endptr,
        int __base, locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 4)));
__extension__
extern long long int strtoll_l (const char *__restrict __nptr,
    char **__restrict __endptr, int __base,
    locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 4)));
__extension__
extern unsigned long long int strtoull_l (const char *__restrict __nptr,
       char **__restrict __endptr,
       int __base, locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 4)));
extern double strtod_l (const char *__restrict __nptr,
   char **__restrict __endptr, locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern float strtof_l (const char *__restrict __nptr,
         char **__restrict __endptr, locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern long double strtold_l (const char *__restrict __nptr,
         char **__restrict __endptr,
         locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
# 316 "/usr/include/stdlib.h" 3 4
extern _Float32 strtof32_l (const char *__restrict __nptr,
       char **__restrict __endptr,
       locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern _Float64 strtof64_l (const char *__restrict __nptr,
       char **__restrict __endptr,
       locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern _Float128 strtof128_l (const char *__restrict __nptr,
         char **__restrict __endptr,
         locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern _Float32x strtof32x_l (const char *__restrict __nptr,
         char **__restrict __endptr,
         locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
extern _Float64x strtof64x_l (const char *__restrict __nptr,
         char **__restrict __endptr,
         locale_t __loc)
     throw () __attribute__ ((__nonnull__ (1, 3)));
# 385 "/usr/include/stdlib.h" 3 4
extern char *l64a (long int __n) throw () ;
extern long int a64l (const char *__s)
     throw () __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
extern "C" {
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
typedef __ino64_t ino64_t;
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
typedef __off64_t off64_t;
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
typedef __useconds_t useconds_t;
typedef __suseconds_t suseconds_t;
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
# 49 "/usr/include/sys/select.h" 3 4
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
extern "C" {
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
}
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 236 "/usr/include/sys/types.h" 3 4
typedef __blkcnt64_t blkcnt64_t;
typedef __fsblkcnt64_t fsblkcnt64_t;
typedef __fsfilcnt64_t fsfilcnt64_t;
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
}
# 395 "/usr/include/stdlib.h" 2 3 4
extern long int random (void) throw ();
extern void srandom (unsigned int __seed) throw ();
extern char *initstate (unsigned int __seed, char *__statebuf,
   size_t __statelen) throw () __attribute__ ((__nonnull__ (2)));
extern char *setstate (char *__statebuf) throw () __attribute__ ((__nonnull__ (1)));
struct random_data
  {
    int32_t *fptr;
    int32_t *rptr;
    int32_t *state;
    int rand_type;
    int rand_deg;
    int rand_sep;
    int32_t *end_ptr;
  };
extern int random_r (struct random_data *__restrict __buf,
       int32_t *__restrict __result) throw () __attribute__ ((__nonnull__ (1, 2)));
extern int srandom_r (unsigned int __seed, struct random_data *__buf)
     throw () __attribute__ ((__nonnull__ (2)));
extern int initstate_r (unsigned int __seed, char *__restrict __statebuf,
   size_t __statelen,
   struct random_data *__restrict __buf)
     throw () __attribute__ ((__nonnull__ (2, 4)));
extern int setstate_r (char *__restrict __statebuf,
         struct random_data *__restrict __buf)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern int rand (void) throw ();
extern void srand (unsigned int __seed) throw ();
extern int rand_r (unsigned int *__seed) throw ();
extern double drand48 (void) throw ();
extern double erand48 (unsigned short int __xsubi[3]) throw () __attribute__ ((__nonnull__ (1)));
extern long int lrand48 (void) throw ();
extern long int nrand48 (unsigned short int __xsubi[3])
     throw () __attribute__ ((__nonnull__ (1)));
extern long int mrand48 (void) throw ();
extern long int jrand48 (unsigned short int __xsubi[3])
     throw () __attribute__ ((__nonnull__ (1)));
extern void srand48 (long int __seedval) throw ();
extern unsigned short int *seed48 (unsigned short int __seed16v[3])
     throw () __attribute__ ((__nonnull__ (1)));
extern void lcong48 (unsigned short int __param[7]) throw () __attribute__ ((__nonnull__ (1)));
struct drand48_data
  {
    unsigned short int __x[3];
    unsigned short int __old_x[3];
    unsigned short int __c;
    unsigned short int __init;
    __extension__ unsigned long long int __a;
  };
extern int drand48_r (struct drand48_data *__restrict __buffer,
        double *__restrict __result) throw () __attribute__ ((__nonnull__ (1, 2)));
extern int erand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        double *__restrict __result) throw () __attribute__ ((__nonnull__ (1, 2)));
extern int lrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern int nrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern int mrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern int jrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern int srand48_r (long int __seedval, struct drand48_data *__buffer)
     throw () __attribute__ ((__nonnull__ (2)));
extern int seed48_r (unsigned short int __seed16v[3],
       struct drand48_data *__buffer) throw () __attribute__ ((__nonnull__ (1, 2)));
extern int lcong48_r (unsigned short int __param[7],
        struct drand48_data *__buffer)
     throw () __attribute__ ((__nonnull__ (1, 2)));
extern void *malloc (size_t __size) throw () __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
     throw () __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
     throw () __attribute__ ((__warn_unused_result__));
extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
     throw () __attribute__ ((__warn_unused_result__));
extern void free (void *__ptr) throw ();
# 1 "/usr/include/alloca.h" 1 3 4
# 24 "/usr/include/alloca.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 25 "/usr/include/alloca.h" 2 3 4
extern "C" {
extern void *alloca (size_t __size) throw ();
}
# 567 "/usr/include/stdlib.h" 2 3 4
extern void *valloc (size_t __size) throw () __attribute__ ((__malloc__)) ;
extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
     throw () __attribute__ ((__nonnull__ (1))) ;
extern void *aligned_alloc (size_t __alignment, size_t __size)
     throw () __attribute__ ((__malloc__)) __attribute__ ((__alloc_size__ (2))) ;
extern void abort (void) throw () __attribute__ ((__noreturn__));
extern int atexit (void (*__func) (void)) throw () __attribute__ ((__nonnull__ (1)));
extern "C++" int at_quick_exit (void (*__func) (void))
     throw () __asm ("at_quick_exit") __attribute__ ((__nonnull__ (1)));
# 607 "/usr/include/stdlib.h" 3 4
extern int on_exit (void (*__func) (int __status, void *__arg), void *__arg)
     throw () __attribute__ ((__nonnull__ (1)));
extern void exit (int __status) throw () __attribute__ ((__noreturn__));
extern void quick_exit (int __status) throw () __attribute__ ((__noreturn__));
extern void _Exit (int __status) throw () __attribute__ ((__noreturn__));
extern char *getenv (const char *__name) throw () __attribute__ ((__nonnull__ (1))) ;
extern char *secure_getenv (const char *__name)
     throw () __attribute__ ((__nonnull__ (1))) ;
extern int putenv (char *__string) throw () __attribute__ ((__nonnull__ (1)));
extern int setenv (const char *__name, const char *__value, int __replace)
     throw () __attribute__ ((__nonnull__ (2)));
extern int unsetenv (const char *__name) throw () __attribute__ ((__nonnull__ (1)));
extern int clearenv (void) throw ();
# 672 "/usr/include/stdlib.h" 3 4
extern char *mktemp (char *__template) throw () __attribute__ ((__nonnull__ (1)));
# 685 "/usr/include/stdlib.h" 3 4
extern int mkstemp (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 695 "/usr/include/stdlib.h" 3 4
extern int mkstemp64 (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 707 "/usr/include/stdlib.h" 3 4
extern int mkstemps (char *__template, int __suffixlen) __attribute__ ((__nonnull__ (1))) ;
# 717 "/usr/include/stdlib.h" 3 4
extern int mkstemps64 (char *__template, int __suffixlen)
     __attribute__ ((__nonnull__ (1))) ;
# 728 "/usr/include/stdlib.h" 3 4
extern char *mkdtemp (char *__template) throw () __attribute__ ((__nonnull__ (1))) ;
# 739 "/usr/include/stdlib.h" 3 4
extern int mkostemp (char *__template, int __flags) __attribute__ ((__nonnull__ (1))) ;
# 749 "/usr/include/stdlib.h" 3 4
extern int mkostemp64 (char *__template, int __flags) __attribute__ ((__nonnull__ (1))) ;
# 759 "/usr/include/stdlib.h" 3 4
extern int mkostemps (char *__template, int __suffixlen, int __flags)
     __attribute__ ((__nonnull__ (1))) ;
# 771 "/usr/include/stdlib.h" 3 4
extern int mkostemps64 (char *__template, int __suffixlen, int __flags)
     __attribute__ ((__nonnull__ (1))) ;
# 781 "/usr/include/stdlib.h" 3 4
extern int system (const char *__command) ;
extern char *canonicalize_file_name (const char *__name)
     throw () __attribute__ ((__nonnull__ (1))) ;
# 797 "/usr/include/stdlib.h" 3 4
extern char *realpath (const char *__restrict __name,
         char *__restrict __resolved) throw () ;
typedef int (*__compar_fn_t) (const void *, const void *);
typedef __compar_fn_t comparison_fn_t;
typedef int (*__compar_d_fn_t) (const void *, const void *, void *);
extern void *bsearch (const void *__key, const void *__base,
        size_t __nmemb, size_t __size, __compar_fn_t __compar)
     __attribute__ ((__nonnull__ (1, 2, 5))) ;
extern void qsort (void *__base, size_t __nmemb, size_t __size,
     __compar_fn_t __compar) __attribute__ ((__nonnull__ (1, 4)));
extern void qsort_r (void *__base, size_t __nmemb, size_t __size,
       __compar_d_fn_t __compar, void *__arg)
  __attribute__ ((__nonnull__ (1, 4)));
extern int abs (int __x) throw () __attribute__ ((__const__)) ;
extern long int labs (long int __x) throw () __attribute__ ((__const__)) ;
__extension__ extern long long int llabs (long long int __x)
     throw () __attribute__ ((__const__)) ;
extern div_t div (int __numer, int __denom)
     throw () __attribute__ ((__const__)) ;
extern ldiv_t ldiv (long int __numer, long int __denom)
     throw () __attribute__ ((__const__)) ;
__extension__ extern lldiv_t lldiv (long long int __numer,
        long long int __denom)
     throw () __attribute__ ((__const__)) ;
# 869 "/usr/include/stdlib.h" 3 4
extern char *ecvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) throw () __attribute__ ((__nonnull__ (3, 4))) ;
extern char *fcvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) throw () __attribute__ ((__nonnull__ (3, 4))) ;
extern char *gcvt (double __value, int __ndigit, char *__buf)
     throw () __attribute__ ((__nonnull__ (3))) ;
extern char *qecvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     throw () __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qfcvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     throw () __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qgcvt (long double __value, int __ndigit, char *__buf)
     throw () __attribute__ ((__nonnull__ (3))) ;
extern int ecvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) throw () __attribute__ ((__nonnull__ (3, 4, 5)));
extern int fcvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) throw () __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qecvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     throw () __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qfcvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     throw () __attribute__ ((__nonnull__ (3, 4, 5)));
extern int mblen (const char *__s, size_t __n) throw ();
extern int mbtowc (wchar_t *__restrict __pwc,
     const char *__restrict __s, size_t __n) throw ();
extern int wctomb (char *__s, wchar_t __wchar) throw ();
extern size_t mbstowcs (wchar_t *__restrict __pwcs,
   const char *__restrict __s, size_t __n) throw ();
extern size_t wcstombs (char *__restrict __s,
   const wchar_t *__restrict __pwcs, size_t __n)
     throw ();
extern int rpmatch (const char *__response) throw () __attribute__ ((__nonnull__ (1))) ;
# 954 "/usr/include/stdlib.h" 3 4
extern int getsubopt (char **__restrict __optionp,
        char *const *__restrict __tokens,
        char **__restrict __valuep)
     throw () __attribute__ ((__nonnull__ (1, 2, 3))) ;
extern int posix_openpt (int __oflag) ;
extern int grantpt (int __fd) throw ();
extern int unlockpt (int __fd) throw ();
extern char *ptsname (int __fd) throw () ;
extern int ptsname_r (int __fd, char *__buf, size_t __buflen)
     throw () __attribute__ ((__nonnull__ (2)));
extern int getpt (void);
extern int getloadavg (double __loadavg[], int __nelem)
     throw () __attribute__ ((__nonnull__ (1)));
# 1010 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/stdlib-float.h" 1 3 4
# 1011 "/usr/include/stdlib.h" 2 3 4
# 1020 "/usr/include/stdlib.h" 3 4
}
# 76 "/usr/include/c++/8.2.1/cstdlib" 2 3
# 1 "/usr/include/c++/8.2.1/bits/std_abs.h" 1 3
# 33 "/usr/include/c++/8.2.1/bits/std_abs.h" 3
       
# 34 "/usr/include/c++/8.2.1/bits/std_abs.h" 3
# 46 "/usr/include/c++/8.2.1/bits/std_abs.h" 3
extern "C++"
{
namespace std __attribute__ ((__visibility__ ("default")))
{
  using ::abs;
  inline long
  abs(long __i) { return __builtin_labs(__i); }
  inline long long
  abs(long long __x) { return __builtin_llabs (__x); }
  inline constexpr double
  abs(double __x)
  { return __builtin_fabs(__x); }
  inline constexpr float
  abs(float __x)
  { return __builtin_fabsf(__x); }
  inline constexpr long double
  abs(long double __x)
  { return __builtin_fabsl(__x); }
  inline constexpr __int128
  abs(__int128 __x) { return __x >= 0 ? __x : -__x; }
# 100 "/usr/include/c++/8.2.1/bits/std_abs.h" 3
  inline constexpr
  __float128
  abs(__float128 __x)
  { return __x < 0 ? -__x : __x; }
}
}
# 78 "/usr/include/c++/8.2.1/cstdlib" 2 3
# 121 "/usr/include/c++/8.2.1/cstdlib" 3
extern "C++"
{
namespace std __attribute__ ((__visibility__ ("default")))
{
  using ::div_t;
  using ::ldiv_t;
  using ::abort;
  using ::atexit;
  using ::at_quick_exit;
  using ::atof;
  using ::atoi;
  using ::atol;
  using ::bsearch;
  using ::calloc;
  using ::div;
  using ::exit;
  using ::free;
  using ::getenv;
  using ::labs;
  using ::ldiv;
  using ::malloc;
  using ::mblen;
  using ::mbstowcs;
  using ::mbtowc;
  using ::qsort;
  using ::quick_exit;
  using ::rand;
  using ::realloc;
  using ::srand;
  using ::strtod;
  using ::strtol;
  using ::strtoul;
  using ::system;
  using ::wcstombs;
  using ::wctomb;
  inline ldiv_t
  div(long __i, long __j) { return ldiv(__i, __j); }
}
# 195 "/usr/include/c++/8.2.1/cstdlib" 3
namespace __gnu_cxx __attribute__ ((__visibility__ ("default")))
{
  using ::lldiv_t;
  using ::_Exit;
  using ::llabs;
  inline lldiv_t
  div(long long __n, long long __d)
  { lldiv_t __q; __q.quot = __n / __d; __q.rem = __n % __d; return __q; }
  using ::lldiv;
# 227 "/usr/include/c++/8.2.1/cstdlib" 3
  using ::atoll;
  using ::strtoll;
  using ::strtoull;
  using ::strtof;
  using ::strtold;
}
namespace std
{
  using ::__gnu_cxx::lldiv_t;
  using ::__gnu_cxx::_Exit;
  using ::__gnu_cxx::llabs;
  using ::__gnu_cxx::div;
  using ::__gnu_cxx::lldiv;
  using ::__gnu_cxx::atoll;
  using ::__gnu_cxx::strtof;
  using ::__gnu_cxx::strtoll;
  using ::__gnu_cxx::strtoull;
  using ::__gnu_cxx::strtold;
}
}
# 3 "/tmp/petsc-VuReVk/config.setCompilers/conftest.cc" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
              Popping language Cxx
================================================================================
TEST checkFortranCompiler from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:970)
TESTING: checkFortranCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:970)
  Locate a functional Fortran compiler
Checking for program /home/ryan/bin/mpif90...not found
Checking for program /usr/local/bin/mpif90...not found
Checking for program /home/ryan/bin/mpif90...not found
Checking for program /usr/local/bin/mpif90...not found
Checking for program /usr/local/sbin/mpif90...not found
Checking for program /usr/bin/mpif90...found
                Defined make macro "FC" to "mpif90"
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
                            Pushing language FC
                            Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-VuReVk/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
                Popping language FC
================================================================================
TEST checkFortranComments from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:991)
TESTING: checkFortranComments from config.setCompilers(config/BuildSystem/config/setCompilers.py:991)
  Make sure fortran comment "!" works
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main
! comment

      end
            Fortran comments can use ! in column 1
              Popping language FC
================================================================================
TEST checkLargeFileIO from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1109)
TESTING: checkLargeFileIO from config.setCompilers(config/BuildSystem/config/setCompilers.py:1109)
================================================================================
TEST checkArchiver from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1208)
TESTING: checkArchiver from config.setCompilers(config/BuildSystem/config/setCompilers.py:1208)
  Check that the archiver exists and can make a library usable by the compiler
              Pushing language C
Executing: ar -V
stdout:
GNU ar (GNU Binutils) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
Executing: ar -V
stdout:
GNU ar (GNU Binutils) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}

Checking for program /home/ryan/bin/ar...not found
Checking for program /usr/local/bin/ar...not found
Checking for program /home/ryan/bin/ar...not found
Checking for program /usr/local/bin/ar...not found
Checking for program /usr/local/sbin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /home/ryan/bin/ranlib...not found
Checking for program /usr/local/bin/ranlib...not found
Checking for program /home/ryan/bin/ranlib...not found
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/local/sbin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib -c"
Executing: /usr/bin/ar cr /tmp/petsc-VuReVk/config.setCompilers/libconf1.a /tmp/petsc-VuReVk/config.setCompilers/conf1.o
Executing: /usr/bin/ranlib -c /tmp/petsc-VuReVk/config.setCompilers/libconf1.a
Possible ERROR while running ranlib:
stderr:
/usr/bin/ranlib: invalid option -- 'c'
            Ranlib is not functional with your archiver.  Try --with-ranlib=true if ranlib is unnecessary.
Executing: ar -V
stdout:
GNU ar (GNU Binutils) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
Executing: ar -V
stdout:
GNU ar (GNU Binutils) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}

Checking for program /home/ryan/bin/ar...not found
Checking for program /usr/local/bin/ar...not found
Checking for program /home/ryan/bin/ar...not found
Checking for program /usr/local/bin/ar...not found
Checking for program /usr/local/sbin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /home/ryan/bin/ranlib...not found
Checking for program /usr/local/bin/ranlib...not found
Checking for program /home/ryan/bin/ranlib...not found
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/local/sbin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib"
Executing: /usr/bin/ar cr /tmp/petsc-VuReVk/config.setCompilers/libconf1.a /tmp/petsc-VuReVk/config.setCompilers/conf1.o
Executing: /usr/bin/ranlib /tmp/petsc-VuReVk/config.setCompilers/libconf1.a
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
extern int foo(int);

int main() {
  int b = foo(1);  if (b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o -L/tmp/petsc-VuReVk/config.setCompilers -lconf1 
              Defined make macro "AR_FLAGS" to "cr"
              Defined make macro "AR_LIB_SUFFIX" to "a"
              Popping language C
================================================================================
TEST checkSharedLinker from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1322)
TESTING: checkSharedLinker from config.setCompilers(config/BuildSystem/config/setCompilers.py:1322)
  Check that the linker can produce shared libraries
Executing: uname -s
stdout: Linux
            Checking shared linker mpicc using flags ['-shared']
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /usr/local/sbin/mpicc...not found
Checking for program /usr/bin/mpicc...found
                Defined make macro "LD_SHARED" to "mpicc"
            Trying C compiler flag 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added C compiler flag 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest  -shared  -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
              Valid C linker flag -shared
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
int (*fprintf_ptr)(FILE*,const char*,...) = fprintf;
void  foo(void){
  fprintf_ptr(stdout,"hello");
  return;
}
void bar(void){foo();}
                                              Pushing language C
                                              Popping language C
                                              Pushing language CUDA
                                              Popping language CUDA
                                              Pushing language Cxx
                                              Popping language Cxx
                                              Pushing language FC
                                              Popping language FC
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(void);

int main() {
int ret = foo();
if (ret) {}
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.o  -L/tmp/petsc-VuReVk/config.setCompilers -lconftest
            Using shared linker mpicc with flags ['-shared'] and library extension so
Executing: uname -s
stdout: Linux
================================================================================
TEST checkPIC from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1068)
TESTING: checkPIC from config.setCompilers(config/BuildSystem/config/setCompilers.py:1068)
  Determine the PIC option for each compiler
              Pushing language C
            Trying C compiler flag 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp     /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added C compiler flag 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
int (*fprintf_ptr)(FILE*,const char*,...) = fprintf;
void  foo(void){
  fprintf_ptr(stdout,"hello");
  return;
}
void bar(void){foo();}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
            Accepted C compiler flag 
              Popping language C
              Pushing language Cxx
            Trying Cxx compiler flag 
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp      /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added Cxx compiler flag 
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp     /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
int (*fprintf_ptr)(FILE*,const char*,...) = fprintf;
void  foo(void){
  fprintf_ptr(stdout,"hello");
  return;
}
void bar(void){foo();}
                  Pushing language C
                      Pushing language C
                      Popping language C
                  Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
            Accepted Cxx compiler flag 
              Popping language Cxx
              Pushing language FC
            Trying FC compiler flag 
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
Added FC compiler flag 
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      function foo(a)
      real:: a,x,bar
      common /xx/ x
      x=a
      foo = bar(x)
      end
                  Pushing language C
                      Pushing language C
                      Popping language C
                  Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
            Accepted FC compiler flag 
              Popping language FC
================================================================================
TEST checkSharedLinkerPaths from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1416)
TESTING: checkSharedLinkerPaths from config.setCompilers(config/BuildSystem/config/setCompilers.py:1416)
  Determine the shared linker path options
       - IRIX: -rpath
       - Linux, OSF: -Wl,-rpath,
       - Solaris: -R
       - FreeBSD: -Wl,-R,
              Pushing language C
Executing: uname -s
stdout: Linux
Executing: mpicc -V
            Trying C linker flag -Wl,-rpath,
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest  -Wl,-rpath,/home/ryan/projects/moose/petsc  -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
              Valid C linker flag -Wl,-rpath,/home/ryan/projects/moose/petsc
              Popping language C
              Pushing language Cxx
Executing: uname -s
stdout: Linux
Executing: mpicc -V
            Trying Cxx linker flag -Wl,-rpath,
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp     /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                      Pushing language CXX
                      Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest  -Wl,-rpath,/home/ryan/projects/moose/petsc  -fPIC -fopenmp /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
              Valid Cxx linker flag -Wl,-rpath,/home/ryan/projects/moose/petsc
              Popping language Cxx
              Pushing language FC
Executing: uname -s
stdout: Linux
Executing: mpicc -V
            Trying FC linker flag -Wl,-rpath,
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.setCompilers/conftest  -Wl,-rpath,/home/ryan/projects/moose/petsc  -fPIC -fopenmp  /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
              Valid FC linker flag -Wl,-rpath,/home/ryan/projects/moose/petsc
              Popping language FC
================================================================================
TEST checkLibC from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1451)
TESTING: checkLibC from config.setCompilers(config/BuildSystem/config/setCompilers.py:1451)
  Test whether we need to explicitly include libc in shared linking
       - Mac OSX requires an explicit reference to libc for shared linking
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdlib.h> 
int foo(void) {void *chunk = malloc(31); free(chunk); return 0;}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o 
            Shared linking does not require an explicit libc reference
================================================================================
TEST checkDynamicLinker from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1500)
TESTING: checkDynamicLinker from config.setCompilers(config/BuildSystem/config/setCompilers.py:1500)
  Check that the linker can dynamicaly load shared libraries
Checking for header: dlfcn.h
                          All intermediate test results are stored in /tmp/petsc-VuReVk/config.headers
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/dlfcn.h" 1 3 4
# 22 "/usr/include/dlfcn.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/dlfcn.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 25 "/usr/include/dlfcn.h" 2 3 4
# 1 "/usr/include/bits/dlfcn.h" 1 3 4
# 28 "/usr/include/dlfcn.h" 2 3 4
# 52 "/usr/include/dlfcn.h" 3 4
extern void *dlopen (const char *__file, int __mode) __attribute__ ((__nothrow__));
extern int dlclose (void *__handle) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern void *dlsym (void *__restrict __handle,
      const char *__restrict __name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 82 "/usr/include/dlfcn.h" 3 4
extern char *dlerror (void) __attribute__ ((__nothrow__ , __leaf__));
# 188 "/usr/include/dlfcn.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_DLFCN_H" to "1"
                Checking for functions [dlopen dlsym dlclose] in library ['dl'] []
                  Pushing language C
                              All intermediate test results are stored in /tmp/petsc-VuReVk/config.libraries
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char dlopen();
static void _check_dlopen() { dlopen(); }
char dlsym();
static void _check_dlsym() { dlsym(); }
char dlclose();
static void _check_dlclose() { dlclose(); }

int main() {
_check_dlopen();
_check_dlsym();
_check_dlclose();;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   /tmp/petsc-VuReVk/config.libraries/conftest.o  -ldl 
                  Defined "HAVE_LIBDL" to "1"
                  Popping language C
              Adding ['dl'] to LIBS
Executing: uname -s
stdout: Linux
            Checking dynamic linker mpicc using flags ['-shared']
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /home/ryan/bin/mpicc...not found
Checking for program /usr/local/bin/mpicc...not found
Checking for program /usr/local/sbin/mpicc...not found
Checking for program /usr/bin/mpicc...found
                Defined make macro "DYNAMICLINKER" to "mpicc"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest  -shared  -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o -ldl 
              Valid C linker flag -shared
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
int foo(void) {printf("test");return 0;}
                                              Pushing language C
                                              Popping language C
                                              Pushing language CUDA
                                              Popping language CUDA
                                              Pushing language Cxx
                                              Popping language Cxx
                                              Pushing language FC
                                              Popping language FC
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/libconftest.so  -shared -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp    /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.setCompilers/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:11:3: warning: implicit declaration of function ???printf??? [-Wimplicit-function-declaration]
   printf("Could not load symbol\n");
   ^~~~~~
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:11:3: warning: incompatible implicit declaration of built-in function ???printf???
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:11:3: note: include ???<stdio.h>??? or provide a declaration of ???printf???
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:4:1:
+#include <stdio.h>
 
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:11:3:
   printf("Could not load symbol\n");
   ^~~~~~
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:15:3: warning: incompatible implicit declaration of built-in function ???printf???
   printf("Invalid return from foo()\n");
   ^~~~~~
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:15:3: note: include ???<stdio.h>??? or provide a declaration of ???printf???
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:19:3: warning: incompatible implicit declaration of built-in function ???printf???
   printf("Could not close library\n");
   ^~~~~~
/tmp/petsc-VuReVk/config.setCompilers/conftest.c:19:3: note: include ???<stdio.h>??? or provide a declaration of ???printf???
Source:
#include "confdefs.h"
#include "conffix.h"
#include<dlfcn.h>

int main() {

void *handle = dlopen("/tmp/petsc-VuReVk/config.setCompilers/libconftest.so", 0);
int (*foo)(void) = (int (*)(void)) dlsym(handle, "foo");

if (!foo) {
  printf("Could not load symbol\n");
  return -1;
}
if ((*foo)()) {
  printf("Invalid return from foo()\n");
  return -1;
}
if (dlclose(handle)) {
  printf("Could not close library\n");
  return -1;
}
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp   /tmp/petsc-VuReVk/config.setCompilers/conftest.o -ldl 
            Using dynamic linker mpicc with flags ['-shared'] and library extension so
================================================================================
TEST output from config.setCompilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/setCompilers.py:1552)
TESTING: output from config.setCompilers(config/BuildSystem/config/setCompilers.py:1552)
  Output module data as defines and substitutions
              Substituting "CC" with "mpicc"
              Substituting "CFLAGS" with "-fPIC -fopenmp  "
              Defined make macro "CC_LINKER_SLFLAG" to "-Wl,-rpath,"
              Substituting "CPP" with "mpicc -E"
              Substituting "CPPFLAGS" with ""
              Substituting "CXX" with "mpicxx"
              Substituting "CXX_CXXFLAGS" with " "
              Substituting "CXXFLAGS" with "-fPIC -fopenmp"
              Substituting "CXX_LINKER_SLFLAG" with "-Wl,-rpath,"
              Substituting "CXXCPP" with "mpicxx -E"
              Substituting "CXXCPPFLAGS" with ""
              Substituting "FC" with "mpif90"
              Substituting "FFLAGS" with "-fPIC -fopenmp "
              Defined make macro "FC_LINKER_SLFLAG" to "-Wl,-rpath,"
              Substituting "LDFLAGS" with ""
              Substituting "LIBS" with "-ldl "
              Substituting "SHARED_LIBRARY_FLAG" with "-shared"
================================================================================
TEST checkSharedDynamicPicOptions from PETSc.options.sharedLibraries(/home/ryan/projects/moose/petsc/config/PETSc/options/sharedLibraries.py:37)
TESTING: checkSharedDynamicPicOptions from PETSc.options.sharedLibraries(config/PETSc/options/sharedLibraries.py:37)
================================================================================
TEST configureSharedLibraries from PETSc.options.sharedLibraries(/home/ryan/projects/moose/petsc/config/PETSc/options/sharedLibraries.py:53)
TESTING: configureSharedLibraries from PETSc.options.sharedLibraries(config/PETSc/options/sharedLibraries.py:53)
  Checks whether shared libraries should be used, for which you must
      - Specify --with-shared-libraries
      - Have found a working shared linker
    Defines PETSC_USE_SHARED_LIBRARIES if they are used
Executing: uname -s
stdout: Linux
              Defined make rule "shared_arch" with dependencies "shared_linux" and code []
              Defined make macro "SONAME_FUNCTION" to "$(1).so.$(2)"
              Defined make macro "SL_LINKER_FUNCTION" to "-shared -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2))"
              Defined make macro "BUILDSHAREDLIB" to "yes"
              Defined "HAVE_SHARED_LIBRARIES" to "1"
              Defined "USE_SHARED_LIBRARIES" to "1"
================================================================================
TEST configureDynamicLibraries from PETSc.options.sharedLibraries(/home/ryan/projects/moose/petsc/config/PETSc/options/sharedLibraries.py:97)
TESTING: configureDynamicLibraries from PETSc.options.sharedLibraries(config/PETSc/options/sharedLibraries.py:97)
  Checks whether dynamic loading is available (with dlfcn.h and libdl)
              Defined "HAVE_DYNAMIC_LIBRARIES" to "1"
================================================================================
TEST configureSerializedFunctions from PETSc.options.sharedLibraries(/home/ryan/projects/moose/petsc/config/PETSc/options/sharedLibraries.py:103)
TESTING: configureSerializedFunctions from PETSc.options.sharedLibraries(config/PETSc/options/sharedLibraries.py:103)
  
    Defines PETSC_SERIALIZE_FUNCTIONS if they are used
    Requires shared libraries
================================================================================
TEST configureIndexSize from PETSc.options.indexTypes(/home/ryan/projects/moose/petsc/config/PETSc/options/indexTypes.py:31)
TESTING: configureIndexSize from PETSc.options.indexTypes(config/PETSc/options/indexTypes.py:31)
              Defined make macro "PETSC_INDEX_SIZE" to "32"
================================================================================
TEST configureCompilerFlags from config.compilerFlags(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilerFlags.py:72)
TESTING: configureCompilerFlags from config.compilerFlags(config/BuildSystem/config/compilerFlags.py:72)
  Get the default compiler flags
            Ignoring default options which were overridden using --CFLAGS -fPIC -fopenmp
                  Defined make macro "MPICC_SHOW" to "gcc -pthread -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi"
            Trying C compiler flag -g
            Trying C compiler flag -O
            Ignoring default options which were overridden using --CXXFLAGS -fPIC -fopenmp
                  Defined make macro "MPICXX_SHOW" to "g++ -pthread -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi_cxx -lmpi"
            Trying Cxx compiler flag -g
            Trying Cxx compiler flag -O
            Ignoring default options which were overridden using --FFLAGS -fPIC -fopenmp
                  Defined make macro "MPIFC_SHOW" to "/usr/bin/gfortran -I/usr/include -pthread -I/usr/lib/openmpi -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi"
            Trying FC compiler flag -g
            Trying FC compiler flag -O
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added C compiler flag -g
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added C compiler flag -O
              Popping language C
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g      /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added Cxx compiler flag -g
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O      /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Added Cxx compiler flag -O
              Popping language Cxx
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g   /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
Added FC compiler flag -g
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
Added FC compiler flag -O
              Popping language FC
Executing: mpicc --version
stdout:
gcc (GCC) 8.2.1 20181127
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
getCompilerVersion: mpicc gcc (GCC) 8.2.1 20181127
Executing: mpicc -show
stdout: gcc -pthread -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi
Executing: mpicc --help
stdout:
Usage: gcc [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gcc.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
Executing: mpicc --help
stdout:
Usage: gcc [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gcc.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
                    ===============================================================================
                        ***** WARNING: Using default optimization C flags -g -O
                        You might consider manually setting optimal optimization flags for your system with
                         COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
                    ===============================================================================
                  
Executing: mpicxx --version
stdout:
g++ (GCC) 8.2.1 20181127
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
getCompilerVersion: mpicxx g++ (GCC) 8.2.1 20181127
Executing: mpicxx -show
stdout: g++ -pthread -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi_cxx -lmpi
Executing: mpicxx --help
stdout:
Usage: g++ [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by g++.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
Executing: mpicxx --help
stdout:
Usage: g++ [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by g++.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
                    ===============================================================================
                        ***** WARNING: Using default C++ optimization flags -g -O
                        You might consider manually setting optimal optimization flags for your system with
                         CXXOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
                    ===============================================================================
                  
Executing: mpif90 --version
stdout:
GNU Fortran (GCC) 8.2.1 20181127
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
getCompilerVersion: mpif90 GNU Fortran (GCC) 8.2.1 20181127
Executing: mpif90 -show
stdout: /usr/bin/gfortran -I/usr/include -pthread -I/usr/lib/openmpi -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
Executing: mpif90 --help
stdout:
Usage: gfortran [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gfortran.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
                    ===============================================================================
                        ***** WARNING: Using default FORTRAN optimization flags -g -O
                        You might consider manually setting optimal optimization flags for your system with
                         FOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
                    ===============================================================================
                  
Executing: mpicc --help
stdout:
Usage: gcc [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gcc.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
================================================================================
TEST checkRestrict from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:139)
TESTING: checkRestrict from config.compilers(config/BuildSystem/config/compilers.py:139)
  Check for the C/CXX restrict keyword
Executing: mpicc -V
              Pushing language C
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.compilers
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
float * __restrict x;;
  return 0;
}
            compilers: Set C restrict keyword to __restrict
              Defined "C_RESTRICT" to "__restrict"
              Popping language C
================================================================================
TEST checkCFormatting from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:391)
TESTING: checkCFormatting from config.compilers(config/BuildSystem/config/compilers.py:391)
  Activate format string checking if using the GNU compilers
================================================================================
TEST checkCInline from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:109)
TESTING: checkCInline from config.compilers(config/BuildSystem/config/compilers.py:109)
  Check for C inline keyword
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
static inline int foo(int a) {return a;}

int main() {
foo(1);;
  return 0;
}
            compilers: Set C Inline keyword to inline
              Popping language C
              Defined "C_INLINE" to "inline"
================================================================================
TEST checkDynamicLoadFlag from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:402)
TESTING: checkDynamicLoadFlag from config.compilers(config/BuildSystem/config/compilers.py:402)
  Checks that dlopen() takes RTLD_XXX, and defines PETSC_HAVE_RTLD_XXX if it does
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <dlfcn.h>
char *libname;

int main() {
dlopen(libname, RTLD_LAZY);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o -ldl 
              Defined "HAVE_RTLD_LAZY" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <dlfcn.h>
char *libname;

int main() {
dlopen(libname, RTLD_NOW);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o -ldl 
              Defined "HAVE_RTLD_NOW" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <dlfcn.h>
char *libname;

int main() {
dlopen(libname, RTLD_LOCAL);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o -ldl 
              Defined "HAVE_RTLD_LOCAL" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <dlfcn.h>
char *libname;

int main() {
dlopen(libname, RTLD_GLOBAL);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o -ldl 
              Defined "HAVE_RTLD_GLOBAL" to "1"
================================================================================
TEST checkCLibraries from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:203)
TESTING: checkCLibraries from config.compilers(config/BuildSystem/config/compilers.py:203)
  Determines the libraries needed to link with C compiled code
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
void asub(void)
{char s[16];printf("testing %s",s);}
                Popping language C
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      print*,'testing'
      stop
      end
                         Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -ldl 
                Popping language FC
            C libraries are not needed when using Fortran linker
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
void asub(void)
{char s[16];printf("testing %s",s);}
                Popping language C
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O     /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
 int main(int argc,char **args)
{return 0;}
                         Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -ldl 
                Popping language Cxx
            C libraries are not needed when using C++ linker
================================================================================
TEST checkDependencyGenerationFlag from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1589)
TESTING: checkDependencyGenerationFlag from config.compilers(config/BuildSystem/config/compilers.py:1589)
  Check if -MMD works for dependency generation, and add it if it does
            Trying C compiler flag -MMD -MP
              Defined make macro "C_DEPFLAGS" to "-MMD -MP"
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O -MMD -MP  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
              Popping language C
            Trying Cxx compiler flag -MMD -MP
              Defined make macro "CXX_DEPFLAGS" to "-MMD -MP"
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O   -MMD -MP   /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
              Popping language Cxx
            Trying FC compiler flag -MMD -MP
              Defined make macro "FC_DEPFLAGS" to "-MMD -MP"
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O -MMD -MP  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
              Popping language FC
================================================================================
TEST checkC99Flag from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1635)
TESTING: checkC99Flag from config.compilers(config/BuildSystem/config/compilers.py:1635)
  Check for -std=c99 or equivalent flag
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <float.h>

int main() {

    float x[2],y;
    y = FLT_ROUNDS;
    // c++ comment
    int j = 2;
    for (int i=0; i<2; i++){
      x[i] = i*j*y;
    }
    ;
  return 0;
}
            Accepted C99 compile flag: 
              Defined "HAVE_C99" to "1"
              Popping language C
Executing: mpicxx --help
stdout:
Usage: g++ [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a dynamically linked position independent
                           executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by g++.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
================================================================================
TEST checkRestrict from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:139)
TESTING: checkRestrict from config.compilers(config/BuildSystem/config/compilers.py:139)
  Check for the C/CXX restrict keyword
Executing: mpicc -V
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O     /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
float * __restrict x;;
  return 0;
}
            compilers: Set Cxx restrict keyword to __restrict
              Defined "CXX_RESTRICT" to "__restrict"
              Popping language Cxx
================================================================================
TEST checkCxxNamespace from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:442)
TESTING: checkCxxNamespace from config.compilers(config/BuildSystem/config/compilers.py:442)
  Checks that C++ compiler supports namespaces, and if it does defines HAVE_CXX_NAMESPACE
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O     /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
namespace petsc {int dummy;}

int main() {
;
  return 0;
}
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O     /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
template <class dummy> struct a {};
namespace trouble{
template <class dummy> struct a : public ::a<dummy> {};
}
trouble::a<int> uugh;

int main() {
;
  return 0;
}
              Popping language Cxx
            compilers: C++ has namespaces
              Defined "HAVE_CXX_NAMESPACE" to "1"
================================================================================
TEST checkCxxOptionalExtensions from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:415)
TESTING: checkCxxOptionalExtensions from config.compilers(config/BuildSystem/config/compilers.py:415)
  Check whether the C++ compiler (IBM xlC, OSF5) need special flag for .c files which contain C++
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
class somename { int i; };;
  return 0;
}
Added Cxx compiler flag 
              Popping language Cxx
================================================================================
TEST checkCxxInline from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:124)
TESTING: checkCxxInline from config.compilers(config/BuildSystem/config/compilers.py:124)
  Check for C++ inline keyword
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O      /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
static inline int foo(int a) {return a;}

int main() {
foo(1);;
  return 0;
}
            compilers: Set Cxx Inline keyword to inline
              Popping language Cxx
              Defined "CXX_INLINE" to "inline"
================================================================================
TEST checkCxxLibraries from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:500)
TESTING: checkCxxLibraries from config.compilers(config/BuildSystem/config/compilers.py:500)
  Determines the libraries needed to link with C++
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O      /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <iostream>
#include <vector>
void asub(void)
{std::vector<int> v;
try  { throw 20;  }  catch (int e)  { std::cout << "An exception occurred";  }}
                Popping language Cxx
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
 int main(int argc,char **args)
{return 0;}
                         Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o: in function `asub()':
/tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `__cxa_allocate_exception'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `typeinfo for int'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `__cxa_throw'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `__cxa_begin_catch'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `std::cout'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `__cxa_end_catch'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.cc:7: undefined reference to `__cxa_end_catch'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o: in function `__static_initialization_and_destruction_0':
/usr/include/c++/8.2.1/iostream:74: undefined reference to `std::ios_base::Init::Init()'
/usr/bin/ld: /usr/include/c++/8.2.1/iostream:74: undefined reference to `std::ios_base::Init::~Init()'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o:(.data.rel.local.DW.ref._ZTIi[DW.ref._ZTIi]+0x0): undefined reference to `typeinfo for int'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o:(.data.rel.local.DW.ref.__gxx_personality_v0[DW.ref.__gxx_personality_v0]+0x0): undefined reference to `__gxx_personality_v0'
collect2: error: ld returned 1 exit status
                Popping language C
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O      /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <iostream>
#include <vector>
void asub(void)
{std::vector<int> v;
try  { throw 20;  }  catch (int e)  { std::cout << "An exception occurred";  }}
                Popping language Cxx
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
 int main(int argc,char **args)
{return 0;}
                         Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -lstdc++ -ldl 
                Popping language C
            compilers: C++ requires -lstdc++ to link with C compiler
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O      /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <iostream>
#include <vector>
void asub(void)
{std::vector<int> v;
try  { throw 20;  }  catch (int e)  { std::cout << "An exception occurred";  }}
                Popping language Cxx
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      print*,'testing'
      stop
      end
                         Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -lstdc++ -ldl 
                Popping language FC
            C++ libraries are not needed when using FC linker
================================================================================
TEST checkCxx11 from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:457)
TESTING: checkCxx11 from config.compilers(config/BuildSystem/config/compilers.py:457)
  Determine the option needed to support the C++11 dialect

    We auto-detect C++11 if the compiler supports it without options,
    otherwise we require with-cxx-dialect=C++11 to try adding flags to
    support it.
    
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

          #include <random>
          template<typename T> constexpr T Cubed( T x ) { return x*x*x; }
          

int main() {

          std::random_device rd;
          std::mt19937 mt(rd());
          std::normal_distribution<double> dist(0,1);
          const double x = dist(mt);
          ;
  return 0;
}
              Popping language Cxx
================================================================================
TEST checkFortranTypeSizes from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:713)
TESTING: checkFortranTypeSizes from config.compilers(config/BuildSystem/config/compilers.py:713)
  Check whether real*8 is supported and suggest flags which will allow support
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      real*8 variable
      end
              Popping language FC
================================================================================
TEST checkFortranNameMangling from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:773)
TESTING: checkFortranNameMangling from config.compilers(config/BuildSystem/config/compilers.py:773)
  Checks Fortran name mangling, and defines HAVE_FORTRAN_UNDERSCORE, HAVE_FORTRAN_NOUNDERSCORE, HAVE_FORTRAN_CAPS, or HAVE_FORTRAN_STDCALL
Testing Fortran mangling type underscore with code void d1chk_(void){return;}
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
void d1chk_(void){return;}
                Popping language C
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
       call d1chk()

      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o  -lstdc++ -ldl 
                Popping language FC
            compilers: Fortran name mangling is underscore
              Defined "HAVE_FORTRAN_UNDERSCORE" to "1"
                    Pushing language FC
                    Popping language FC
Executing: mpif90 --version
stdout:
GNU Fortran (GCC) 8.2.1 20181127
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
              Defined "FORTRAN_CHARLEN_T" to "size_t"
================================================================================
TEST checkFortranNameManglingDouble from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:814)
TESTING: checkFortranNameManglingDouble from config.compilers(config/BuildSystem/config/compilers.py:814)
  Checks if symbols containing an underscore append an extra underscore, and defines HAVE_FORTRAN_UNDERSCORE_UNDERSCORE if necessary
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
void d1_chk__(void){return;}
                Popping language C
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
       call d1_chk()

      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o  -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.o: in function `MAIN__':
/tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `d1_chk_'
collect2: error: ld returned 1 exit status
                Popping language FC
================================================================================
TEST checkFortranPreprocessor from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:824)
TESTING: checkFortranPreprocessor from config.compilers(config/BuildSystem/config/compilers.py:824)
  Determine if Fortran handles preprocessing properly
            compilers: Fortran uses CPP preprocessor
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O    /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main
#define dummy 
           dummy
#ifndef dummy
       fooey
#endif
      end
Added FC compiler flag 
              Popping language FC
================================================================================
TEST checkFortranDefineCompilerOption from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:848)
TESTING: checkFortranDefineCompilerOption from config.compilers(config/BuildSystem/config/compilers.py:848)
  Check if -WF,-Dfoobar or -Dfoobar is the compiler option to define a macro
              Defined make macro "FC_DEFINE_FLAG" to "-D"
            compilers: Fortran uses -D for defining macro
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  -DTesting  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main
#define dummy 
           dummy
#ifndef Testing
       fooey
#endif
      end
              Popping language FC
================================================================================
TEST checkFortranLibraries from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:868)
TESTING: checkFortranLibraries from config.compilers(config/BuildSystem/config/compilers.py:868)
  Substitutes for FLIBS the libraries needed to link with Fortran

    This macro is intended to be used in those situations when it is
    necessary to mix, e.g. C++ and Fortran 77, source code into a single
    program or shared library.

    For example, if object files from a C++ and Fortran 77 compiler must
    be linked together, then the C++ compiler/linker must be used for
    linking (since special C++-ish things need to happen at link time
    like calling global constructors, instantiating templates, enabling
    exception support, etc.).

    However, the Fortran 77 intrinsic and run-time libraries must be
    linked in as well, but the C++ compiler/linker does not know how to
    add these Fortran 77 libraries.

    This code was translated from the autoconf macro which was packaged in
    its current form by Matthew D. Langston <langston at SLAC.Stanford.EDU>.
    However, nearly all of this macro came from the OCTAVE_FLIBS macro in
    octave-2.0.13/aclocal.m4, and full credit should go to John W. Eaton
    for writing this extremely useful macro.
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
#include <mpif.h>
      call MPI_Allreduce()

      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl 
              Popping language FC
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
subroutine asub()
      print*,'testing'
      call MPI_Allreduce()
      return
      end
                Popping language FC
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
 int main(int argc,char **args)
{return 0;}
                         Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o: in function `asub_':
/tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_st_write'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_transfer_character_write'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_st_write_done'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:3: undefined reference to `mpi_allreduce_'
collect2: error: ld returned 1 exit status
                Popping language C
            Fortran code cannot directly be linked with C linker, therefor will determine needed Fortran libraries
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
subroutine asub()
      print*,'testing'
      call MPI_Allreduce()
      return
      end
                Popping language FC
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
 int main(int argc,char **args)
{return 0;}
                         Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/confc.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/confc.o: in function `asub_':
/tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_st_write'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_transfer_character_write'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:2: undefined reference to `_gfortran_st_write_done'
/usr/bin/ld: /tmp/petsc-VuReVk/config.compilers/conftest.F90:3: undefined reference to `mpi_allreduce_'
collect2: error: ld returned 1 exit status
                Popping language Cxx
            Fortran code cannot directly be linked with C++ linker, therefor will determine needed Fortran libraries
              Pushing language FC
Executing: mpif90 -V
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main

      end
                    Pushing language FC
                    Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest  -v  -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker:
stderr:
Driving: /usr/bin/gfortran -o /tmp/petsc-VuReVk/config.compilers/conftest -v -fPIC -fopenmp -g -O /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl -I/usr/include -pthread -I/usr/lib/openmpi -Wl,-rpath -Wl,/usr/lib/openmpi -Wl,--enable-new-dtags -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -l gfortran -l m -shared-libgcc
Using built-in specs.
COLLECT_GCC=/usr/bin/gfortran
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --enable-cet=auto
Thread model: posix
gcc version 8.2.1 20181127 (GCC) 
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgfortran.spec
rename spec lib to liborig
COLLECT_GCC_OPTIONS='-o' '/tmp/petsc-VuReVk/config.compilers/conftest' '-v' '-fPIC' '-fopenmp' '-g' '-O' '-I' '/usr/include' '-pthread' '-I' '/usr/lib/openmpi' '-L/usr/lib/openmpi' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
COMPILER_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../:/lib/:/usr/lib/
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgomp.spec
COLLECT_GCC_OPTIONS='-o' '/tmp/petsc-VuReVk/config.compilers/conftest' '-v' '-fPIC' '-fopenmp' '-g' '-O' '-I' '/usr/include' '-pthread' '-I' '/usr/lib/openmpi' '-L/usr/lib/openmpi' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
 /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/collect2 -plugin /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/ccqf7wFt.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o /tmp/petsc-VuReVk/config.compilers/conftest /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/Scrt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtbeginS.o -L/usr/lib/openmpi -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../.. /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl -rpath /usr/lib/openmpi --enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgomp -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crtn.o
COLLECT_GCC_OPTIONS='-o' '/tmp/petsc-VuReVk/config.compilers/conftest' '-v' '-fPIC' '-fopenmp' '-g' '-O' '-I' '/usr/include' '-pthread' '-I' '/usr/lib/openmpi' '-L/usr/lib/openmpi' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
              Popping language FC
            compilers: Checking arg Driving:
            compilers: Unknown arg Driving:
            compilers: Checking arg /usr/bin/gfortran
            compilers: Unknown arg /usr/bin/gfortran
            compilers: Checking arg -o
            compilers: Unknown arg -o
            compilers: Checking arg /tmp/petsc-VuReVk/config.compilers/conftest
            compilers: Unknown arg /tmp/petsc-VuReVk/config.compilers/conftest
            compilers: Checking arg -v
            compilers: Unknown arg -v
            compilers: Checking arg -fPIC
            compilers: Unknown arg -fPIC
            compilers: Checking arg -fopenmp
            compilers: Unknown arg -fopenmp
            compilers: Checking arg -g
            compilers: Unknown arg -g
            compilers: Checking arg -O
            compilers: Unknown arg -O
            compilers: Checking arg /tmp/petsc-VuReVk/config.compilers/conftest.o
            compilers: Unknown arg /tmp/petsc-VuReVk/config.compilers/conftest.o
            compilers: Checking arg -lstdc++
            compilers: Found library: -lstdc++
            compilers: Checking arg -ldl
            compilers: Found library: -ldl
            compilers: Checking arg -I/usr/include
            compilers: Found include directory: /usr/include
            compilers: Checking arg -pthread
            compilers: Unknown arg -pthread
            compilers: Checking arg -I/usr/lib/openmpi
            compilers: Found include directory: /usr/lib/openmpi
            compilers: Checking arg -Wl,-rpath
            compilers: Unknown arg -Wl,-rpath
            compilers: Checking arg -Wl,/usr/lib/openmpi
            compilers: Unknown arg -Wl,/usr/lib/openmpi
            compilers: Checking arg -Wl,--enable-new-dtags
            compilers: Unknown arg -Wl,--enable-new-dtags
            compilers: Checking arg -L/usr/lib/openmpi
            compilers: Found library directory: -L/usr/lib/openmpi
            compilers: Checking arg -lmpi_usempif08
            compilers: Found library: -lmpi_usempif08
            compilers: Checking arg -lmpi_usempi_ignore_tkr
            compilers: Found library: -lmpi_usempi_ignore_tkr
            compilers: Checking arg -lmpi_mpifh
            compilers: Found library: -lmpi_mpifh
            compilers: Checking arg -lmpi
            compilers: Found library: -lmpi
            compilers: Checking arg -l
            compilers: Found canonical library: -lgfortran
            compilers: Checking arg -l
            compilers: Found canonical library: -lm
            compilers: Checking arg -shared-libgcc
            compilers: Unknown arg -shared-libgcc
            compilers: Checking arg Using
            compilers: Unknown arg Using
            compilers: Checking arg built-in
            compilers: Unknown arg built-in
            compilers: Checking arg specs.
            compilers: Unknown arg specs.
            compilers: Checking arg COLLECT_GCC=/usr/bin/gfortran
            compilers: Unknown arg COLLECT_GCC=/usr/bin/gfortran
            compilers: Checking arg COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper
            compilers: Unknown arg COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper
            compilers: Checking arg Target:
            compilers: Unknown arg Target:
            compilers: Checking arg x86_64-pc-linux-gnu
            compilers: Unknown arg x86_64-pc-linux-gnu
            compilers: Checking arg Configured
            compilers: Unknown arg Configured
            compilers: Checking arg with:
            compilers: Unknown arg with:
            compilers: Checking arg /build/gcc/src/gcc/configure
            compilers: Unknown arg /build/gcc/src/gcc/configure
            compilers: Checking arg --prefix=/usr
            compilers: Unknown arg --prefix=/usr
            compilers: Checking arg --libdir=/usr/lib
            compilers: Unknown arg --libdir=/usr/lib
            compilers: Checking arg --libexecdir=/usr/lib
            compilers: Unknown arg --libexecdir=/usr/lib
            compilers: Checking arg --mandir=/usr/share/man
            compilers: Unknown arg --mandir=/usr/share/man
            compilers: Checking arg --infodir=/usr/share/info
            compilers: Unknown arg --infodir=/usr/share/info
            compilers: Checking arg --with-bugurl=https://bugs.archlinux.org/
            compilers: Unknown arg --with-bugurl=https://bugs.archlinux.org/
            compilers: Checking arg --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++
            compilers: Unknown arg --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++
            compilers: Checking arg --enable-shared
            compilers: Unknown arg --enable-shared
            compilers: Checking arg --enable-threads=posix
            compilers: Unknown arg --enable-threads=posix
            compilers: Checking arg --enable-libmpx
            compilers: Unknown arg --enable-libmpx
            compilers: Checking arg --with-system-zlib
            compilers: Unknown arg --with-system-zlib
            compilers: Checking arg --with-isl
            compilers: Unknown arg --with-isl
            compilers: Checking arg --enable-__cxa_atexit
            compilers: Unknown arg --enable-__cxa_atexit
            compilers: Checking arg --disable-libunwind-exceptions
            compilers: Unknown arg --disable-libunwind-exceptions
            compilers: Checking arg --enable-clocale=gnu
            compilers: Unknown arg --enable-clocale=gnu
            compilers: Checking arg --disable-libstdcxx-pch
            compilers: Unknown arg --disable-libstdcxx-pch
            compilers: Checking arg --disable-libssp
            compilers: Unknown arg --disable-libssp
            compilers: Checking arg --enable-gnu-unique-object
            compilers: Unknown arg --enable-gnu-unique-object
            compilers: Checking arg --enable-linker-build-id
            compilers: Unknown arg --enable-linker-build-id
            compilers: Checking arg --enable-lto
            compilers: Unknown arg --enable-lto
            compilers: Checking arg --enable-plugin
            compilers: Unknown arg --enable-plugin
            compilers: Checking arg --enable-install-libiberty
            compilers: Unknown arg --enable-install-libiberty
            compilers: Checking arg --with-linker-hash-style=gnu
            compilers: Unknown arg --with-linker-hash-style=gnu
            compilers: Checking arg --enable-gnu-indirect-function
            compilers: Unknown arg --enable-gnu-indirect-function
            compilers: Checking arg --enable-multilib
            compilers: Unknown arg --enable-multilib
            compilers: Checking arg --disable-werror
            compilers: Unknown arg --disable-werror
            compilers: Checking arg --enable-checking=release
            compilers: Unknown arg --enable-checking=release
            compilers: Checking arg --enable-default-pie
            compilers: Unknown arg --enable-default-pie
            compilers: Checking arg --enable-default-ssp
            compilers: Unknown arg --enable-default-ssp
            compilers: Checking arg --enable-cet=auto
            compilers: Unknown arg --enable-cet=auto
            compilers: Checking arg Thread
            compilers: Unknown arg Thread
            compilers: Checking arg model:
            compilers: Unknown arg model:
            compilers: Checking arg posix
            compilers: Unknown arg posix
            compilers: Checking arg gcc
            compilers: Unknown arg gcc
            compilers: Checking arg version
            compilers: Unknown arg version
            compilers: Checking arg 8.2.1
            compilers: Unknown arg 8.2.1
            compilers: Checking arg 20181127
            compilers: Unknown arg 20181127
            compilers: Checking arg (GCC)
            compilers: Unknown arg (GCC)
            compilers: Checking arg Reading
            compilers: Unknown arg Reading
            compilers: Checking arg specs
            compilers: Unknown arg specs
            compilers: Checking arg from
            compilers: Unknown arg from
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgfortran.spec
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgfortran.spec
            compilers: Checking arg rename
            compilers: Unknown arg rename
            compilers: Checking arg spec
            compilers: Unknown arg spec
            compilers: Checking arg lib
            compilers: Unknown arg lib
            compilers: Checking arg to
            compilers: Unknown arg to
            compilers: Checking arg liborig
            compilers: Unknown arg liborig
            compilers: Checking arg COLLECT_GCC_OPTIONS=
            compilers: Unknown arg COLLECT_GCC_OPTIONS=
            compilers: Checking arg COMPILER_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/
            compilers: Skipping arg COMPILER_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/
            compilers: Checking arg LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../:/lib/:/usr/lib/
            compilers: Skipping arg LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../:/lib/:/usr/lib/
            compilers: Checking arg Reading
            compilers: Unknown arg Reading
            compilers: Checking arg specs
            compilers: Unknown arg specs
            compilers: Checking arg from
            compilers: Unknown arg from
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgomp.spec
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/libgomp.spec
            compilers: Checking arg COLLECT_GCC_OPTIONS=
            compilers: Unknown arg COLLECT_GCC_OPTIONS=
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/collect2
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/collect2
            compilers: Checking arg -plugin
            compilers: Unknown arg -plugin
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/liblto_plugin.so
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/liblto_plugin.so
            compilers: Checking arg -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper
            compilers: Unknown arg -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/lto-wrapper
            compilers: Checking arg -plugin-opt=-fresolution=/tmp/ccqf7wFt.res
            compilers: Unknown arg -plugin-opt=-fresolution=/tmp/ccqf7wFt.res
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc
            compilers: Checking arg -plugin-opt=-pass-through=-lquadmath
            compilers: Unknown arg -plugin-opt=-pass-through=-lquadmath
            compilers: Checking arg -plugin-opt=-pass-through=-lm
            compilers: Unknown arg -plugin-opt=-pass-through=-lm
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc
            compilers: Checking arg -plugin-opt=-pass-through=-lpthread
            compilers: Unknown arg -plugin-opt=-pass-through=-lpthread
            compilers: Checking arg -plugin-opt=-pass-through=-lc
            compilers: Unknown arg -plugin-opt=-pass-through=-lc
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc_s
            compilers: Checking arg -plugin-opt=-pass-through=-lgcc
            compilers: Unknown arg -plugin-opt=-pass-through=-lgcc
            compilers: Checking arg --build-id
            compilers: Unknown arg --build-id
            compilers: Checking arg --eh-frame-hdr
            compilers: Unknown arg --eh-frame-hdr
            compilers: Checking arg --hash-style=gnu
            compilers: Unknown arg --hash-style=gnu
            compilers: Checking arg -m
            compilers: Unknown arg -m
            compilers: Checking arg elf_x86_64
            compilers: Unknown arg elf_x86_64
            compilers: Checking arg -dynamic-linker
            compilers: Unknown arg -dynamic-linker
            compilers: Checking arg /lib64/ld-linux-x86-64.so.2
            compilers: Unknown arg /lib64/ld-linux-x86-64.so.2
            compilers: Checking arg -pie
            compilers: Unknown arg -pie
            compilers: Checking arg -o
            compilers: Unknown arg -o
            compilers: Checking arg /tmp/petsc-VuReVk/config.compilers/conftest
            compilers: Unknown arg /tmp/petsc-VuReVk/config.compilers/conftest
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/Scrt1.o
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/Scrt1.o
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crti.o
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crti.o
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtbeginS.o
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtbeginS.o
            compilers: Checking arg -L/usr/lib/openmpi
            compilers: Already in lflags so skipping: -L/usr/lib/openmpi
            compilers: Checking arg -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1
            compilers: Found library directory: -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1
            compilers: Checking arg -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib
            compilers: Checking arg -L/lib/../lib
            compilers: Checking arg -L/usr/lib/../lib
            compilers: Checking arg -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../..
            compilers: Checking arg /tmp/petsc-VuReVk/config.compilers/conftest.o
            compilers: Unknown arg /tmp/petsc-VuReVk/config.compilers/conftest.o
            compilers: Checking arg -lstdc++
            compilers: Already in lflags: -lstdc++
            compilers: Checking arg -ldl
            compilers: Already in lflags: -ldl
            compilers: Checking arg -rpath
            compilers: Found -rpath library: /usr/lib/openmpi
            compilers: Checking arg --enable-new-dtags
            compilers: Unknown arg --enable-new-dtags
            compilers: Checking arg -lmpi_usempif08
            compilers: Already in lflags: -lmpi_usempif08
            compilers: Checking arg -lmpi_usempi_ignore_tkr
            compilers: Already in lflags: -lmpi_usempi_ignore_tkr
            compilers: Checking arg -lmpi_mpifh
            compilers: Already in lflags: -lmpi_mpifh
            compilers: Checking arg -lmpi
            compilers: Already in lflags: -lmpi
            compilers: Checking arg -lgfortran
            compilers: Found library: -lgfortran
            compilers: Checking arg -lm
            compilers: Found library: -lm
            compilers: Checking arg -lgomp
            compilers: Found library: -lgomp
            compilers: Checking arg -lgcc_s
            compilers: Found library: -lgcc_s
            compilers: Checking arg -lgcc
            compilers: Found system library therefor skipping: -lgcc
            compilers: Checking arg -lquadmath
            compilers: Found library: -lquadmath
            compilers: Checking arg -lm
            compilers: Already in lflags: -lm
            compilers: Checking arg -lgcc_s
            compilers: Already in lflags: -lgcc_s
            compilers: Checking arg -lgcc
            compilers: Found system library therefor skipping: -lgcc
            compilers: Checking arg -lpthread
            compilers: Found library: -lpthread
            compilers: Checking arg -lc
            compilers: Found system library therefor skipping: -lc
            compilers: Checking arg -lgcc_s
            compilers: Already in lflags: -lgcc_s
            compilers: Checking arg -lgcc
            compilers: Found system library therefor skipping: -lgcc
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtendS.o
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/crtendS.o
            compilers: Checking arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crtn.o
            compilers: Unknown arg /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/crtn.o
            compilers: Checking arg COLLECT_GCC_OPTIONS=
            compilers: Unknown arg COLLECT_GCC_OPTIONS=
            compilers: Libraries needed to link Fortran code with the C linker: ['-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
            compilers: Libraries needed to link Fortran main with the C linker: []
            compilers: Check that Fortran libraries can be used from C
                        Pushing language C
                        Popping language C
                        Pushing language C
                        Popping language C
                            Pushing language C
                            Popping language C
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
                Popping language C
            compilers: Check that Fortran libraries can be used from C++
                        Pushing language CXX
                        Popping language CXX
                        Pushing language CXX
                        Popping language CXX
                            Pushing language CXX
                            Popping language CXX
            compilers: Fortran libraries can be used from C++
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.setCompilers/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.setCompilers/conftest.o -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
Executing: /tmp/petsc-VuReVk/config.setCompilers/conftest
                Popping language Cxx
================================================================================
TEST checkFortranLinkingCxx from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1263)
TESTING: checkFortranLinkingCxx from config.compilers(config/BuildSystem/config/compilers.py:1263)
  Check that Fortran can be linked against C++
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
extern "C" void d1chk_(void);
void foo(void){d1chk_();}
              Popping language Cxx
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.compilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
extern "C" void d1chk_(void);
void d1chk_(void){return;}
                Popping language Cxx
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
       call d1chk()

      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/cxxobj.o  /tmp/petsc-VuReVk/config.compilers/confc.o  -lstdc++ -ldl 
                Popping language FC
            compilers: Fortran can link C++ functions
================================================================================
TEST checkFortran90 from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1329)
TESTING: checkFortran90 from config.compilers(config/BuildSystem/config/compilers.py:1329)
  Determine whether the Fortran compiler handles F90
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      INTEGER, PARAMETER :: int = SELECTED_INT_KIND(8)
      INTEGER (KIND=int) :: ierr

      ierr = 1
      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl 
              Defined "USING_F90" to "1"
            Fortran compiler supports F90
              Popping language FC
================================================================================
TEST checkFortran90FreeForm from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1342)
TESTING: checkFortran90FreeForm from config.compilers(config/BuildSystem/config/compilers.py:1342)
  Determine whether the Fortran compiler handles F90FreeForm
       We also require that the compiler handles lines longer than 132 characters
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.compilers/conftest.F90:2:132:

       INTEGER, PARAMETER ::        int = SELECTED_INT_KIND(8);              INTEGER (KIND=int) :: ierr;       ierr                            =                                                                                                               1
                                                                                                                                    1
Error: Line truncated at (1) [-Werror=line-truncation]
/tmp/petsc-VuReVk/config.compilers/conftest.F90:2:110:

       INTEGER, PARAMETER ::        int = SELECTED_INT_KIND(8);              INTEGER (KIND=int) :: ierr;       ierr                            =                                                                                                               1
                                                                                                              1
Error: Unclassifiable statement at (1)
f951: some warnings being treated as errors
Source:
      program main
      INTEGER, PARAMETER ::        int = SELECTED_INT_KIND(8);              INTEGER (KIND=int) :: ierr;       ierr                            =                                                                                                               1
      end
                Compile failed inside link

            Fortran compiler does not support F90FreeForm
              Popping language FC
================================================================================
TEST checkFortran2003 from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1356)
TESTING: checkFortran2003 from config.compilers(config/BuildSystem/config/compilers.py:1356)
  Determine whether the Fortran compiler handles F2003
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:

      module Base_module
        type, public :: base_type
           integer :: A
         contains
           procedure, public :: Print => BasePrint
        end type base_type
      contains
        subroutine BasePrint(this)
          class(base_type) :: this
        end subroutine BasePrint
      end module Base_module

      program main
      use,intrinsic :: iso_c_binding
      Type(C_Ptr),Dimension(:),Pointer :: CArray
      character(kind=c_char),pointer   :: nullc => null()
      character(kind=c_char,len=5),dimension(:),pointer::list1

      allocate(list1(5))
      CArray = (/(c_loc(list1(i)),i=1,5),c_loc(nullc)/)
      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl 
              Defined "USING_F2003" to "1"
            Fortran compiler supports F2003
              Popping language FC
================================================================================
TEST checkFortran90Array from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1391)
TESTING: checkFortran90Array from config.compilers(config/BuildSystem/config/compilers.py:1391)
  Check for F90 array interfaces
Executing: uname -s
stdout: Linux
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include<stdio.h>
#include <stdlib.h>
void f90arraytest_(void* a1, void* a2,void* a3, void* i)
{
  printf("arrays [%p %p %p]\n",a1,a2,a3);
  fflush(stdout);
  return;
}
void f90ptrtest_(void* a1, void* a2,void* a3, void* i, void* p1 ,void* p2, void* p3)
{
  printf("arrays [%p %p %p]\n",a1,a2,a3);
  if ((p1 == p3) && (p1 != p2)) {
    printf("pointers match! [%p %p] [%p]\n",p1,p3,p2);
    fflush(stdout);
  } else {
    printf("pointers do not match! [%p %p] [%p]\n",p1,p3,p2);
    fflush(stdout);
    exit(111);
  }
  return;
}
              Popping language C
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      Interface
         Subroutine f90ptrtest(p1,p2,p3,i)
         integer, pointer :: p1(:,:)
         integer, pointer :: p2(:,:)
         integer, pointer :: p3(:,:)
         integer i
         End Subroutine
      End Interface

      integer, pointer :: ptr1(:,:),ptr2(:,:)
      integer, target  :: array(6:8,9:21)
      integer  in

      in   = 25
      ptr1 => array
      ptr2 => array

      call f90arraytest(ptr1,ptr2,ptr1,in)
      call f90ptrtest(ptr1,ptr2,ptr1,in)

      end
                          Pushing language FC
                          Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/fooobj.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.compilers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.compilers/conftest
Executing: /tmp/petsc-VuReVk/config.compilers/conftest
stdout:
arrays [0x7ffdfd5d4e00 0x7ffdfd5d4e00 0x7ffdfd5d4e00]
arrays [0x7ffdfd5d4d40 0x7ffdfd5d4da0 0x7ffdfd5d4d40]
pointers do not match! [0x7f89c02d8720 (nil)] [0x7f89bfcedf00]
ERROR while running executable: Could not execute "/tmp/petsc-VuReVk/config.compilers/conftest":
arrays [0x7ffdfd5d4e00 0x7ffdfd5d4e00 0x7ffdfd5d4e00]
arrays [0x7ffdfd5d4d40 0x7ffdfd5d4da0 0x7ffdfd5d4d40]
pointers do not match! [0x7f89c02d8720 (nil)] [0x7f89bfcedf00]
              Popping language FC
            compilers: F90 uses a single argument for array pointers
================================================================================
TEST checkFortranModuleInclude from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1478)
TESTING: checkFortranModuleInclude from config.compilers(config/BuildSystem/config/compilers.py:1478)
  Figures out what flag is used to specify the include path for Fortran modules
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      module configtest
      integer testint
      parameter (testint = 42)
      end module configtest
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      use configtest

      write(*,*) testint

      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -I/tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o /tmp/petsc-VuReVk/config.compilers/configtest.o -lstdc++ -ldl 
            compilers: Fortran module include flag -I found
              Popping language FC
================================================================================
TEST checkFortranModuleOutput from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1544)
TESTING: checkFortranModuleOutput from config.compilers(config/BuildSystem/config/compilers.py:1544)
  Figures out what flag is used to specify the include path for Fortran modules
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -module /tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Possible ERROR while running compiler: exit code 256
stderr:
gfortran: error: unrecognized command line option ???-module???; did you mean ???-mhle????
Source:
      module configtest
      integer testint
      parameter (testint = 42)
      end module configtest
            compilers: Fortran module output flag -module  compile failed
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -module:/tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Possible ERROR while running compiler: exit code 256
stderr:
gfortran: error: unrecognized command line option ???-module:/tmp/petsc-VuReVk/config.compilers/confdir???
Source:
      module configtest
      integer testint
      parameter (testint = 42)
      end module configtest
            compilers: Fortran module output flag -module: compile failed
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fmod=/tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Possible ERROR while running compiler: exit code 256
stderr:
gfortran: error: unrecognized command line option ???-fmod=/tmp/petsc-VuReVk/config.compilers/confdir???
Source:
      module configtest
      integer testint
      parameter (testint = 42)
      end module configtest
            compilers: Fortran module output flag -fmod= compile failed
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -J/tmp/petsc-VuReVk/config.compilers/confdir -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      module configtest
      integer testint
      parameter (testint = 42)
      end module configtest
            compilers: Fortran module output flag -J found
              Popping language FC
================================================================================
TEST checkFortranTypeStar from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1318)
TESTING: checkFortranTypeStar from config.compilers(config/BuildSystem/config/compilers.py:1318)
  Determine whether the Fortran compiler handles type(*)
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      interface
      subroutine a(b)
      type(*) :: b(:)
      end subroutine
      end interface

      end
              Defined "HAVE_FORTRAN_TYPE_STAR" to "1"
            Fortran compiler supports type(*)
              Popping language FC
================================================================================
TEST checkFortranTypeInitialize from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1307)
TESTING: checkFortranTypeInitialize from config.compilers(config/BuildSystem/config/compilers.py:1307)
  Determines if PETSc objects in Fortran are initialized by default (doesn't work with common blocks)
              Defined "HAVE_FORTRAN_TYPE_INITIALIZE" to "-2"
              Defined "FORTRAN_TYPE_INITIALIZE" to " = -2"
            Initializing Fortran objects
================================================================================
TEST configureFortranFlush from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1298)
TESTING: configureFortranFlush from config.compilers(config/BuildSystem/config/compilers.py:1298)
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.compilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.compilers/conftest.F90 
Successful compile:
Source:
      program main
      call flush(6)
      end
                      Pushing language FC
                      Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.compilers/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.compilers/conftest.o -lstdc++ -ldl 
              Defined "HAVE_FORTRAN_FLUSH" to "1"
              Popping language FC
================================================================================
TEST setupFrameworkCompilers from config.compilers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/compilers.py:1711)
TESTING: setupFrameworkCompilers from config.compilers(config/BuildSystem/config/compilers.py:1711)
================================================================================
TEST configureClosure from config.utilities.closure(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/closure.py:18)
TESTING: configureClosure from config.utilities.closure(config/BuildSystem/config/utilities/closure.py:18)
  Determine if Apple ^close syntax is supported in C
              Pushing language C
                          All intermediate test results are stored in /tmp/petsc-VuReVk/config.utilities.closure
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.closure/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.closure/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.utilities.closure/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.utilities.closure/conftest.c:6:6: error: expected identifier or ???(??? before ???^??? token
 int (^closure)(int);;
      ^
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>

int main() {
int (^closure)(int);;
  return 0;
}
                Compile failed inside link

================================================================================
TEST configureFortranCPP from PETSc.options.fortranCPP(/home/ryan/projects/moose/petsc/config/PETSc/options/fortranCPP.py:21)
TESTING: configureFortranCPP from PETSc.options.fortranCPP(config/PETSc/options/fortranCPP.py:21)
  Handle case where Fortran cannot preprocess properly
              Defined make rule ".f.o .f90.o .f95.o" with dependencies "" and code ['${PETSC_MAKE_STOP_ON_ERROR}${FC} -c ${FC_FLAGS} ${FFLAGS} -o $@ $<']
              Defined make rule ".f.a" with dependencies "" and code ['${PETSC_MAKE_STOP_ON_ERROR}${FC} -c ${FC_FLAGS} ${FFLAGS} $<', '-${AR} ${AR_FLAGS} ${LIBNAME} $*.o', '-${RM} $*.o']
              Defined make rule ".F.o .F90.o .F95.o" with dependencies "" and code ['${PETSC_MAKE_STOP_ON_ERROR}${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS} -o $@ $<']
              Defined make rule ".F.a" with dependencies "" and code ['${PETSC_MAKE_STOP_ON_ERROR}${FC} -c ${FC_FLAGS} ${FFLAGS} ${FCPPFLAGS} $<', '-${AR} ${AR_FLAGS} ${LIBNAME} $*.o', '-${RM} $*.o']
================================================================================
TEST checkStdC from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:105)
TESTING: checkStdC from config.headers(config/BuildSystem/config/headers.py:105)
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <float.h>

int main() {
;
  return 0;
}
Source:
#include "confdefs.h"
#include "conffix.h"
#include <string.h>
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/string.h" 1 3 4
# 26 "/usr/include/string.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 27 "/usr/include/string.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 34 "/usr/include/string.h" 2 3 4
# 42 "/usr/include/string.h" 3 4
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
       size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memmove (void *__dest, const void *__src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memccpy (void *__restrict __dest, const void *__restrict __src,
        int __c, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memset (void *__s, int __c, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int memcmp (const void *__s1, const void *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 90 "/usr/include/string.h" 3 4
extern void *memchr (const void *__s, int __c, size_t __n)
      __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 121 "/usr/include/string.h" 3 4
extern char *strcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strncpy (char *__restrict __dest,
        const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strcat (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strncat (char *__restrict __dest, const char *__restrict __src,
        size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcmp (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strncmp (const char *__s1, const char *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcoll (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern size_t strxfrm (char *__restrict __dest,
         const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 153 "/usr/include/string.h" 2 3 4
extern int strcoll_l (const char *__s1, const char *__s2, locale_t __l)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern size_t strxfrm_l (char *__dest, const char *__src, size_t __n,
    locale_t __l) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern char *strdup (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__nonnull__ (1)));
extern char *strndup (const char *__string, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__nonnull__ (1)));
# 225 "/usr/include/string.h" 3 4
extern char *strchr (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 252 "/usr/include/string.h" 3 4
extern char *strrchr (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 272 "/usr/include/string.h" 3 4
extern size_t strcspn (const char *__s, const char *__reject)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern size_t strspn (const char *__s, const char *__accept)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 302 "/usr/include/string.h" 3 4
extern char *strpbrk (const char *__s, const char *__accept)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 329 "/usr/include/string.h" 3 4
extern char *strstr (const char *__haystack, const char *__needle)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strtok (char *__restrict __s, const char *__restrict __delim)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *__strtok_r (char *__restrict __s,
    const char *__restrict __delim,
    char **__restrict __save_ptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern char *strtok_r (char *__restrict __s, const char *__restrict __delim,
         char **__restrict __save_ptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
# 384 "/usr/include/string.h" 3 4
extern size_t strlen (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern size_t strnlen (const char *__string, size_t __maxlen)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern char *strerror (int __errnum) __attribute__ ((__nothrow__ , __leaf__));
# 409 "/usr/include/string.h" 3 4
extern int strerror_r (int __errnum, char *__buf, size_t __buflen) __asm__ ("" "__xpg_strerror_r") __attribute__ ((__nothrow__ , __leaf__))
                        __attribute__ ((__nonnull__ (2)));
# 427 "/usr/include/string.h" 3 4
extern char *strerror_l (int __errnum, locale_t __l) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/strings.h" 1 3 4
# 23 "/usr/include/strings.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 24 "/usr/include/strings.h" 2 3 4
extern int bcmp (const void *__s1, const void *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bcopy (const void *__src, void *__dest, size_t __n)
  __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bzero (void *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 68 "/usr/include/strings.h" 3 4
extern char *index (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 96 "/usr/include/strings.h" 3 4
extern char *rindex (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern int ffs (int __i) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int ffsl (long int __l) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
__extension__ extern int ffsll (long long int __ll)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int strcasecmp (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strncasecmp (const char *__s1, const char *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcasecmp_l (const char *__s1, const char *__s2, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern int strncasecmp_l (const char *__s1, const char *__s2,
     size_t __n, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 4)));
# 432 "/usr/include/string.h" 2 3 4
extern void explicit_bzero (void *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern char *strsep (char **__restrict __stringp,
       const char *__restrict __delim)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strsignal (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern char *__stpcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *stpcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *__stpncpy (char *__restrict __dest,
   const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *stpncpy (char *__restrict __dest,
        const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
# 498 "/usr/include/string.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdlib.h>
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/stdlib.h" 1 3 4
# 25 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 26 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 32 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 40 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 41 "/usr/include/stdlib.h" 2 3 4
# 55 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 56 "/usr/include/stdlib.h" 2 3 4
typedef struct
  {
    int quot;
    int rem;
  } div_t;
typedef struct
  {
    long int quot;
    long int rem;
  } ldiv_t;
__extension__ typedef struct
  {
    long long int quot;
    long long int rem;
  } lldiv_t;
# 97 "/usr/include/stdlib.h" 3 4
extern size_t __ctype_get_mb_cur_max (void) __attribute__ ((__nothrow__ , __leaf__)) ;
extern double atof (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern int atoi (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern long int atol (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
__extension__ extern long long int atoll (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern double strtod (const char *__restrict __nptr,
        char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern float strtof (const char *__restrict __nptr,
       char **__restrict __endptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long double strtold (const char *__restrict __nptr,
       char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 176 "/usr/include/stdlib.h" 3 4
extern long int strtol (const char *__restrict __nptr,
   char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern unsigned long int strtoul (const char *__restrict __nptr,
      char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoq (const char *__restrict __nptr,
        char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtouq (const char *__restrict __nptr,
           char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoll (const char *__restrict __nptr,
         char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtoull (const char *__restrict __nptr,
     char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 385 "/usr/include/stdlib.h" 3 4
extern char *l64a (long int __n) __attribute__ ((__nothrow__ , __leaf__)) ;
extern long int a64l (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 395 "/usr/include/stdlib.h" 2 3 4
extern long int random (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srandom (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern char *initstate (unsigned int __seed, char *__statebuf,
   size_t __statelen) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *setstate (char *__statebuf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct random_data
  {
    int32_t *fptr;
    int32_t *rptr;
    int32_t *state;
    int rand_type;
    int rand_deg;
    int rand_sep;
    int32_t *end_ptr;
  };
extern int random_r (struct random_data *__restrict __buf,
       int32_t *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srandom_r (unsigned int __seed, struct random_data *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int initstate_r (unsigned int __seed, char *__restrict __statebuf,
   size_t __statelen,
   struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern int setstate_r (char *__restrict __statebuf,
         struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int rand (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srand (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern int rand_r (unsigned int *__seed) __attribute__ ((__nothrow__ , __leaf__));
extern double drand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern double erand48 (unsigned short int __xsubi[3]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int lrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int nrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int mrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int jrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void srand48 (long int __seedval) __attribute__ ((__nothrow__ , __leaf__));
extern unsigned short int *seed48 (unsigned short int __seed16v[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void lcong48 (unsigned short int __param[7]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct drand48_data
  {
    unsigned short int __x[3];
    unsigned short int __old_x[3];
    unsigned short int __c;
    unsigned short int __init;
    __extension__ unsigned long long int __a;
  };
extern int drand48_r (struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int erand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int nrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int mrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int jrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srand48_r (long int __seedval, struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int seed48_r (unsigned short int __seed16v[3],
       struct drand48_data *__buffer) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lcong48_r (unsigned short int __param[7],
        struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *malloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
# 563 "/usr/include/stdlib.h" 3 4
extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/alloca.h" 1 3 4
# 24 "/usr/include/alloca.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 25 "/usr/include/alloca.h" 2 3 4
extern void *alloca (size_t __size) __attribute__ ((__nothrow__ , __leaf__));
# 567 "/usr/include/stdlib.h" 2 3 4
extern void *valloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern void *aligned_alloc (size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__alloc_size__ (2))) ;
extern void abort (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern int atexit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int at_quick_exit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int on_exit (void (*__func) (int __status, void *__arg), void *__arg)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void quick_exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void _Exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern char *getenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 644 "/usr/include/stdlib.h" 3 4
extern int putenv (char *__string) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int setenv (const char *__name, const char *__value, int __replace)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int unsetenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int clearenv (void) __attribute__ ((__nothrow__ , __leaf__));
# 672 "/usr/include/stdlib.h" 3 4
extern char *mktemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 685 "/usr/include/stdlib.h" 3 4
extern int mkstemp (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 707 "/usr/include/stdlib.h" 3 4
extern int mkstemps (char *__template, int __suffixlen) __attribute__ ((__nonnull__ (1))) ;
# 728 "/usr/include/stdlib.h" 3 4
extern char *mkdtemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 781 "/usr/include/stdlib.h" 3 4
extern int system (const char *__command) ;
# 797 "/usr/include/stdlib.h" 3 4
extern char *realpath (const char *__restrict __name,
         char *__restrict __resolved) __attribute__ ((__nothrow__ , __leaf__)) ;
typedef int (*__compar_fn_t) (const void *, const void *);
# 817 "/usr/include/stdlib.h" 3 4
extern void *bsearch (const void *__key, const void *__base,
        size_t __nmemb, size_t __size, __compar_fn_t __compar)
     __attribute__ ((__nonnull__ (1, 2, 5))) ;
extern void qsort (void *__base, size_t __nmemb, size_t __size,
     __compar_fn_t __compar) __attribute__ ((__nonnull__ (1, 4)));
# 837 "/usr/include/stdlib.h" 3 4
extern int abs (int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern long int labs (long int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern long long int llabs (long long int __x)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern div_t div (int __numer, int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern ldiv_t ldiv (long int __numer, long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern lldiv_t lldiv (long long int __numer,
        long long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
# 869 "/usr/include/stdlib.h" 3 4
extern char *ecvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *fcvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *gcvt (double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern char *qecvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qfcvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qgcvt (long double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern int ecvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int fcvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qecvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qfcvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int mblen (const char *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int mbtowc (wchar_t *__restrict __pwc,
     const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int wctomb (char *__s, wchar_t __wchar) __attribute__ ((__nothrow__ , __leaf__));
extern size_t mbstowcs (wchar_t *__restrict __pwcs,
   const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern size_t wcstombs (char *__restrict __s,
   const wchar_t *__restrict __pwcs, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__));
extern int rpmatch (const char *__response) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 954 "/usr/include/stdlib.h" 3 4
extern int getsubopt (char **__restrict __optionp,
        char *const *__restrict __tokens,
        char **__restrict __valuep)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3))) ;
# 1000 "/usr/include/stdlib.h" 3 4
extern int getloadavg (double __loadavg[], int __nelem)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1010 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/stdlib-float.h" 1 3 4
# 1011 "/usr/include/stdlib.h" 2 3 4
# 1020 "/usr/include/stdlib.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <stdlib.h>
#include <ctype.h>
#define ISLOWER(c) ('a' <= (c) && (c) <= 'z')
#define TOUPPER(c) (ISLOWER(c) ? 'A' + ((c) - 'a') : (c))
#define XOR(e, f) (((e) && !(f)) || (!(e) && (f)))

int main() {

        int i;

        for(i = 0; i < 256; i++) if (XOR(islower(i), ISLOWER(i)) || toupper(i) != TOUPPER(i)) exit(2);
        exit(0);
      ;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.headers/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.headers/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.headers/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.headers/conftest
Executing: /tmp/petsc-VuReVk/config.headers/conftest
              Defined "STDC_HEADERS" to "1"
================================================================================
TEST checkStat from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:138)
TESTING: checkStat from config.headers(config/BuildSystem/config/headers.py:138)
  Checks whether stat file-mode macros are broken, and defines STAT_MACROS_BROKEN if they are
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#include <sys/stat.h>

#if defined(S_ISBLK) && defined(S_IFDIR)
# if S_ISBLK (S_IFDIR)
  You lose.
# endif
#endif

#if defined(S_ISBLK) && defined(S_IFCHR)
# if S_ISBLK (S_IFCHR)
  You lose.
# endif
#endif

#if defined(S_ISLNK) && defined(S_IFREG)
# if S_ISLNK (S_IFREG)
  You lose.
# endif
#endif

#if defined(S_ISSOCK) && defined(S_IFREG)
# if S_ISSOCK (S_IFREG)
  You lose.
# endif
#endif
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/types.h" 1 3 4
# 25 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 5 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/stat.h" 1 3 4
# 99 "/usr/include/sys/stat.h" 3 4
# 1 "/usr/include/bits/stat.h" 1 3 4
# 46 "/usr/include/bits/stat.h" 3 4
struct stat
  {
    __dev_t st_dev;
    __ino_t st_ino;
    __nlink_t st_nlink;
    __mode_t st_mode;
    __uid_t st_uid;
    __gid_t st_gid;
    int __pad0;
    __dev_t st_rdev;
    __off_t st_size;
    __blksize_t st_blksize;
    __blkcnt_t st_blocks;
# 91 "/usr/include/bits/stat.h" 3 4
    struct timespec st_atim;
    struct timespec st_mtim;
    struct timespec st_ctim;
# 106 "/usr/include/bits/stat.h" 3 4
    __syscall_slong_t __glibc_reserved[3];
# 115 "/usr/include/bits/stat.h" 3 4
  };
# 102 "/usr/include/sys/stat.h" 2 3 4
# 205 "/usr/include/sys/stat.h" 3 4
extern int stat (const char *__restrict __file,
   struct stat *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int fstat (int __fd, struct stat *__buf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 234 "/usr/include/sys/stat.h" 3 4
extern int fstatat (int __fd, const char *__restrict __file,
      struct stat *__restrict __buf, int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
# 259 "/usr/include/sys/stat.h" 3 4
extern int lstat (const char *__restrict __file,
    struct stat *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
# 280 "/usr/include/sys/stat.h" 3 4
extern int chmod (const char *__file, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int lchmod (const char *__file, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int fchmod (int __fd, __mode_t __mode) __attribute__ ((__nothrow__ , __leaf__));
extern int fchmodat (int __fd, const char *__file, __mode_t __mode,
       int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2))) ;
extern __mode_t umask (__mode_t __mask) __attribute__ ((__nothrow__ , __leaf__));
# 317 "/usr/include/sys/stat.h" 3 4
extern int mkdir (const char *__path, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int mkdirat (int __fd, const char *__path, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int mknod (const char *__path, __mode_t __mode, __dev_t __dev)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int mknodat (int __fd, const char *__path, __mode_t __mode,
      __dev_t __dev) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int mkfifo (const char *__path, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int mkfifoat (int __fd, const char *__path, __mode_t __mode)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int utimensat (int __fd, const char *__path,
        const struct timespec __times[2],
        int __flags)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int futimens (int __fd, const struct timespec __times[2]) __attribute__ ((__nothrow__ , __leaf__));
# 395 "/usr/include/sys/stat.h" 3 4
extern int __fxstat (int __ver, int __fildes, struct stat *__stat_buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3)));
extern int __xstat (int __ver, const char *__filename,
      struct stat *__stat_buf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern int __lxstat (int __ver, const char *__filename,
       struct stat *__stat_buf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern int __fxstatat (int __ver, int __fildes, const char *__filename,
         struct stat *__stat_buf, int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4)));
# 438 "/usr/include/sys/stat.h" 3 4
extern int __xmknod (int __ver, const char *__path, __mode_t __mode,
       __dev_t *__dev) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern int __xmknodat (int __ver, int __fd, const char *__path,
         __mode_t __mode, __dev_t *__dev)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 5)));
# 534 "/usr/include/sys/stat.h" 3 4
# 6 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
================================================================================
TEST checkSysWait from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:173)
TESTING: checkSysWait from config.headers(config/BuildSystem/config/headers.py:173)
  Check for POSIX.1 compatible sys/wait.h, and defines HAVE_SYS_WAIT_H if found
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#include <sys/wait.h>
#ifndef WEXITSTATUS
#define WEXITSTATUS(stat_val) ((unsigned)(stat_val) >> 8)
#endif
#ifndef WIFEXITED
#define WIFEXITED(stat_val) (((stat_val) & 255) == 0)
#endif

int main() {

    int s;
    wait (&s);
    s = WIFEXITED (s) ? WEXITSTATUS (s) : 1;
    ;
  return 0;
}
              Defined "HAVE_SYS_WAIT_H" to "1"
================================================================================
TEST checkTime from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:195)
TESTING: checkTime from config.headers(config/BuildSystem/config/headers.py:195)
  Checks if you can safely include both <sys/time.h> and <time.h>, and if so defines TIME_WITH_SYS_TIME
Checking for header: time.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/time.h" 1 3 4
# 25 "/usr/include/time.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/time.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 30 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/time.h" 1 3 4
# 26 "/usr/include/bits/time.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 27 "/usr/include/bits/time.h" 2 3 4
# 34 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 38 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 39 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_tm.h" 1 3 4
struct tm
{
  int tm_sec;
  int tm_min;
  int tm_hour;
  int tm_mday;
  int tm_mon;
  int tm_year;
  int tm_wday;
  int tm_yday;
  int tm_isdst;
  long int tm_gmtoff;
  const char *tm_zone;
};
# 40 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 43 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 47 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 48 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_itimerspec.h" 1 3 4
struct itimerspec
  {
    struct timespec it_interval;
    struct timespec it_value;
  };
# 49 "/usr/include/time.h" 2 3 4
struct sigevent;
typedef __pid_t pid_t;
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 61 "/usr/include/time.h" 2 3 4
extern clock_t clock (void) __attribute__ ((__nothrow__ , __leaf__));
extern time_t time (time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern double difftime (time_t __time1, time_t __time0)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern time_t mktime (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern size_t strftime (char *__restrict __s, size_t __maxsize,
   const char *__restrict __format,
   const struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
# 104 "/usr/include/time.h" 3 4
extern size_t strftime_l (char *__restrict __s, size_t __maxsize,
     const char *__restrict __format,
     const struct tm *__restrict __tp,
     locale_t __loc) __attribute__ ((__nothrow__ , __leaf__));
# 119 "/usr/include/time.h" 3 4
extern struct tm *gmtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *gmtime_r (const time_t *__restrict __timer,
       struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime_r (const time_t *__restrict __timer,
          struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime (const struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime_r (const struct tm *__restrict __tp,
   char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime_r (const time_t *__restrict __timer,
        char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *__tzname[2];
extern int __daylight;
extern long int __timezone;
extern char *tzname[2];
extern void tzset (void) __attribute__ ((__nothrow__ , __leaf__));
extern int daylight;
extern long int timezone;
extern int stime (const time_t *__when) __attribute__ ((__nothrow__ , __leaf__));
# 196 "/usr/include/time.h" 3 4
extern time_t timegm (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern time_t timelocal (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int dysize (int __year) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 211 "/usr/include/time.h" 3 4
extern int nanosleep (const struct timespec *__requested_time,
        struct timespec *__remaining);
extern int clock_getres (clockid_t __clock_id, struct timespec *__res) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_gettime (clockid_t __clock_id, struct timespec *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_settime (clockid_t __clock_id, const struct timespec *__tp)
     __attribute__ ((__nothrow__ , __leaf__));
extern int clock_nanosleep (clockid_t __clock_id, int __flags,
       const struct timespec *__req,
       struct timespec *__rem);
extern int clock_getcpuclockid (pid_t __pid, clockid_t *__clock_id) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_create (clockid_t __clock_id,
    struct sigevent *__restrict __evp,
    timer_t *__restrict __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_delete (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_settime (timer_t __timerid, int __flags,
     const struct itimerspec *__restrict __value,
     struct itimerspec *__restrict __ovalue) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_gettime (timer_t __timerid, struct itimerspec *__value)
     __attribute__ ((__nothrow__ , __leaf__));
extern int timer_getoverrun (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timespec_get (struct timespec *__ts, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 307 "/usr/include/time.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_TIME_H" to "1"
Checking for header: sys/time.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/time.h" 1 3 4
# 21 "/usr/include/sys/time.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/sys/time.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 24 "/usr/include/sys/time.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 25 "/usr/include/sys/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 26 "/usr/include/sys/time.h" 2 3 4
typedef __suseconds_t suseconds_t;
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
# 49 "/usr/include/sys/select.h" 3 4
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 33 "/usr/include/sys/time.h" 2 3 4
# 52 "/usr/include/sys/time.h" 3 4
struct timezone
  {
    int tz_minuteswest;
    int tz_dsttime;
  };
typedef struct timezone *__restrict __timezone_ptr_t;
# 68 "/usr/include/sys/time.h" 3 4
extern int gettimeofday (struct timeval *__restrict __tv,
    __timezone_ptr_t __tz) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int settimeofday (const struct timeval *__tv,
    const struct timezone *__tz)
     __attribute__ ((__nothrow__ , __leaf__));
extern int adjtime (const struct timeval *__delta,
      struct timeval *__olddelta) __attribute__ ((__nothrow__ , __leaf__));
enum __itimer_which
  {
    ITIMER_REAL = 0,
    ITIMER_VIRTUAL = 1,
    ITIMER_PROF = 2
  };
struct itimerval
  {
    struct timeval it_interval;
    struct timeval it_value;
  };
typedef int __itimer_which_t;
extern int getitimer (__itimer_which_t __which,
        struct itimerval *__value) __attribute__ ((__nothrow__ , __leaf__));
extern int setitimer (__itimer_which_t __which,
        const struct itimerval *__restrict __new,
        struct itimerval *__restrict __old) __attribute__ ((__nothrow__ , __leaf__));
extern int utimes (const char *__file, const struct timeval __tvp[2])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int lutimes (const char *__file, const struct timeval __tvp[2])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int futimes (int __fd, const struct timeval __tvp[2]) __attribute__ ((__nothrow__ , __leaf__));
# 186 "/usr/include/sys/time.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_TIME_H" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/types.h>
#include <sys/time.h>
#include <time.h>

int main() {
struct tm *tp = 0;

if (tp);
;
  return 0;
}
              Defined "TIME_WITH_SYS_TIME" to "1"
================================================================================
TEST checkMath from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:203)
TESTING: checkMath from config.headers(config/BuildSystem/config/headers.py:203)
  Checks for the math headers and defines
Checking for header: math.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/math.h" 1 3 4
# 27 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 28 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 38 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/math-vector.h" 1 3 4
# 25 "/usr/include/bits/math-vector.h" 3 4
# 1 "/usr/include/bits/libm-simd-decl-stubs.h" 1 3 4
# 26 "/usr/include/bits/math-vector.h" 2 3 4
# 41 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 44 "/usr/include/math.h" 2 3 4
# 138 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/flt-eval-method.h" 1 3 4
# 139 "/usr/include/math.h" 2 3 4
# 149 "/usr/include/math.h" 3 4
typedef float float_t;
typedef double double_t;
# 190 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/fp-logb.h" 1 3 4
# 191 "/usr/include/math.h" 2 3 4
# 233 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/fp-fast.h" 1 3 4
# 234 "/usr/include/math.h" 2 3 4
# 289 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/mathcalls-helper-functions.h" 1 3 4
# 21 "/usr/include/bits/mathcalls-helper-functions.h" 3 4
extern int __fpclassify (double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __signbit (double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __isinf (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __finite (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __isnan (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __iseqsig (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern int __issignaling (double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
# 290 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/mathcalls.h" 1 3 4
# 53 "/usr/include/bits/mathcalls.h" 3 4
extern double acos (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __acos (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double asin (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __asin (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double atan (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __atan (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double atan2 (double __y, double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __atan2 (double __y, double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern double cos (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __cos (double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern double sin (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __sin (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double tan (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __tan (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double cosh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __cosh (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double sinh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __sinh (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double tanh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __tanh (double __x) __attribute__ ((__nothrow__ , __leaf__));
# 85 "/usr/include/bits/mathcalls.h" 3 4
extern double acosh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __acosh (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double asinh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __asinh (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double atanh (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __atanh (double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern double exp (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __exp (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double frexp (double __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__)); extern double __frexp (double __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__));
extern double ldexp (double __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__)); extern double __ldexp (double __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__));
 extern double log (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __log (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double log10 (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __log10 (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double modf (double __x, double *__iptr) __attribute__ ((__nothrow__ , __leaf__)); extern double __modf (double __x, double *__iptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 119 "/usr/include/bits/mathcalls.h" 3 4
extern double expm1 (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __expm1 (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double log1p (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __log1p (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double logb (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __logb (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double exp2 (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __exp2 (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double log2 (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __log2 (double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern double pow (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __pow (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double sqrt (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __sqrt (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double hypot (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __hypot (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double cbrt (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __cbrt (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double ceil (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __ceil (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double fabs (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __fabs (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double floor (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __floor (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double fmod (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __fmod (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
# 177 "/usr/include/bits/mathcalls.h" 3 4
extern int isinf (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int finite (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double drem (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __drem (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double significand (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __significand (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double copysign (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __copysign (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double nan (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__)); extern double __nan (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__));
# 211 "/usr/include/bits/mathcalls.h" 3 4
extern int isnan (double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double j0 (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __j0 (double) __attribute__ ((__nothrow__ , __leaf__));
extern double j1 (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __j1 (double) __attribute__ ((__nothrow__ , __leaf__));
extern double jn (int, double) __attribute__ ((__nothrow__ , __leaf__)); extern double __jn (int, double) __attribute__ ((__nothrow__ , __leaf__));
extern double y0 (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __y0 (double) __attribute__ ((__nothrow__ , __leaf__));
extern double y1 (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __y1 (double) __attribute__ ((__nothrow__ , __leaf__));
extern double yn (int, double) __attribute__ ((__nothrow__ , __leaf__)); extern double __yn (int, double) __attribute__ ((__nothrow__ , __leaf__));
extern double erf (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __erf (double) __attribute__ ((__nothrow__ , __leaf__));
extern double erfc (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __erfc (double) __attribute__ ((__nothrow__ , __leaf__));
extern double lgamma (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __lgamma (double) __attribute__ ((__nothrow__ , __leaf__));
extern double tgamma (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __tgamma (double) __attribute__ ((__nothrow__ , __leaf__));
extern double gamma (double) __attribute__ ((__nothrow__ , __leaf__)); extern double __gamma (double) __attribute__ ((__nothrow__ , __leaf__));
extern double lgamma_r (double, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__)); extern double __lgamma_r (double, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__));
extern double rint (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __rint (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double nextafter (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __nextafter (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double nexttoward (double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __nexttoward (double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/bits/mathcalls.h" 3 4
extern double remainder (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __remainder (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double scalbn (double __x, int __n) __attribute__ ((__nothrow__ , __leaf__)); extern double __scalbn (double __x, int __n) __attribute__ ((__nothrow__ , __leaf__));
extern int ilogb (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern int __ilogb (double __x) __attribute__ ((__nothrow__ , __leaf__));
# 290 "/usr/include/bits/mathcalls.h" 3 4
extern double scalbln (double __x, long int __n) __attribute__ ((__nothrow__ , __leaf__)); extern double __scalbln (double __x, long int __n) __attribute__ ((__nothrow__ , __leaf__));
extern double nearbyint (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern double __nearbyint (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double round (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __round (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double trunc (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __trunc (double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double remquo (double __x, double __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__)); extern double __remquo (double __x, double __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__));
extern long int lrint (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lrint (double __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llrint (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llrint (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long int lround (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lround (double __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llround (double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llround (double __x) __attribute__ ((__nothrow__ , __leaf__));
extern double fdim (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)); extern double __fdim (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__));
extern double fmax (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __fmax (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double fmin (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern double __fmin (double __x, double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern double fma (double __x, double __y, double __z) __attribute__ ((__nothrow__ , __leaf__)); extern double __fma (double __x, double __y, double __z) __attribute__ ((__nothrow__ , __leaf__));
# 396 "/usr/include/bits/mathcalls.h" 3 4
extern double scalb (double __x, double __n) __attribute__ ((__nothrow__ , __leaf__)); extern double __scalb (double __x, double __n) __attribute__ ((__nothrow__ , __leaf__));
# 291 "/usr/include/math.h" 2 3 4
# 306 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/mathcalls-helper-functions.h" 1 3 4
# 21 "/usr/include/bits/mathcalls-helper-functions.h" 3 4
extern int __fpclassifyf (float __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __signbitf (float __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __isinff (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __finitef (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __isnanf (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __iseqsigf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern int __issignalingf (float __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
# 307 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/mathcalls.h" 1 3 4
# 53 "/usr/include/bits/mathcalls.h" 3 4
extern float acosf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __acosf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float asinf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __asinf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float atanf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __atanf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float atan2f (float __y, float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __atan2f (float __y, float __x) __attribute__ ((__nothrow__ , __leaf__));
 extern float cosf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __cosf (float __x) __attribute__ ((__nothrow__ , __leaf__));
 extern float sinf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __sinf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float tanf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __tanf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float coshf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __coshf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float sinhf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __sinhf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float tanhf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __tanhf (float __x) __attribute__ ((__nothrow__ , __leaf__));
# 85 "/usr/include/bits/mathcalls.h" 3 4
extern float acoshf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __acoshf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float asinhf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __asinhf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float atanhf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __atanhf (float __x) __attribute__ ((__nothrow__ , __leaf__));
 extern float expf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __expf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float frexpf (float __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__)); extern float __frexpf (float __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__));
extern float ldexpf (float __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__)); extern float __ldexpf (float __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__));
 extern float logf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __logf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float log10f (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __log10f (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float modff (float __x, float *__iptr) __attribute__ ((__nothrow__ , __leaf__)); extern float __modff (float __x, float *__iptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 119 "/usr/include/bits/mathcalls.h" 3 4
extern float expm1f (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __expm1f (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float log1pf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __log1pf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float logbf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __logbf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float exp2f (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __exp2f (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float log2f (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __log2f (float __x) __attribute__ ((__nothrow__ , __leaf__));
 extern float powf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __powf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float sqrtf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __sqrtf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float hypotf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __hypotf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float cbrtf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __cbrtf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float ceilf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __ceilf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float fabsf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __fabsf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float floorf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __floorf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float fmodf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __fmodf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
# 177 "/usr/include/bits/mathcalls.h" 3 4
extern int isinff (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int finitef (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float dremf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __dremf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float significandf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __significandf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float copysignf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __copysignf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float nanf (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__)); extern float __nanf (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__));
# 211 "/usr/include/bits/mathcalls.h" 3 4
extern int isnanf (float __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float j0f (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __j0f (float) __attribute__ ((__nothrow__ , __leaf__));
extern float j1f (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __j1f (float) __attribute__ ((__nothrow__ , __leaf__));
extern float jnf (int, float) __attribute__ ((__nothrow__ , __leaf__)); extern float __jnf (int, float) __attribute__ ((__nothrow__ , __leaf__));
extern float y0f (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __y0f (float) __attribute__ ((__nothrow__ , __leaf__));
extern float y1f (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __y1f (float) __attribute__ ((__nothrow__ , __leaf__));
extern float ynf (int, float) __attribute__ ((__nothrow__ , __leaf__)); extern float __ynf (int, float) __attribute__ ((__nothrow__ , __leaf__));
extern float erff (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __erff (float) __attribute__ ((__nothrow__ , __leaf__));
extern float erfcf (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __erfcf (float) __attribute__ ((__nothrow__ , __leaf__));
extern float lgammaf (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __lgammaf (float) __attribute__ ((__nothrow__ , __leaf__));
extern float tgammaf (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __tgammaf (float) __attribute__ ((__nothrow__ , __leaf__));
extern float gammaf (float) __attribute__ ((__nothrow__ , __leaf__)); extern float __gammaf (float) __attribute__ ((__nothrow__ , __leaf__));
extern float lgammaf_r (float, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__)); extern float __lgammaf_r (float, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__));
extern float rintf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __rintf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float nextafterf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __nextafterf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float nexttowardf (float __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __nexttowardf (float __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/bits/mathcalls.h" 3 4
extern float remainderf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __remainderf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float scalbnf (float __x, int __n) __attribute__ ((__nothrow__ , __leaf__)); extern float __scalbnf (float __x, int __n) __attribute__ ((__nothrow__ , __leaf__));
extern int ilogbf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern int __ilogbf (float __x) __attribute__ ((__nothrow__ , __leaf__));
# 290 "/usr/include/bits/mathcalls.h" 3 4
extern float scalblnf (float __x, long int __n) __attribute__ ((__nothrow__ , __leaf__)); extern float __scalblnf (float __x, long int __n) __attribute__ ((__nothrow__ , __leaf__));
extern float nearbyintf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern float __nearbyintf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float roundf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __roundf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float truncf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __truncf (float __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float remquof (float __x, float __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__)); extern float __remquof (float __x, float __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__));
extern long int lrintf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lrintf (float __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llrintf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llrintf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern long int lroundf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lroundf (float __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llroundf (float __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llroundf (float __x) __attribute__ ((__nothrow__ , __leaf__));
extern float fdimf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)); extern float __fdimf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__));
extern float fmaxf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __fmaxf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float fminf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern float __fminf (float __x, float __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern float fmaf (float __x, float __y, float __z) __attribute__ ((__nothrow__ , __leaf__)); extern float __fmaf (float __x, float __y, float __z) __attribute__ ((__nothrow__ , __leaf__));
# 396 "/usr/include/bits/mathcalls.h" 3 4
extern float scalbf (float __x, float __n) __attribute__ ((__nothrow__ , __leaf__)); extern float __scalbf (float __x, float __n) __attribute__ ((__nothrow__ , __leaf__));
# 308 "/usr/include/math.h" 2 3 4
# 349 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/mathcalls-helper-functions.h" 1 3 4
# 21 "/usr/include/bits/mathcalls-helper-functions.h" 3 4
extern int __fpclassifyl (long double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __signbitl (long double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __isinfl (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __finitel (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __isnanl (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __iseqsigl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern int __issignalingl (long double __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
# 350 "/usr/include/math.h" 2 3 4
# 1 "/usr/include/bits/mathcalls.h" 1 3 4
# 53 "/usr/include/bits/mathcalls.h" 3 4
extern long double acosl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __acosl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double asinl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __asinl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double atanl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __atanl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double atan2l (long double __y, long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __atan2l (long double __y, long double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern long double cosl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __cosl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern long double sinl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __sinl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double tanl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __tanl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double coshl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __coshl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double sinhl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __sinhl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double tanhl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __tanhl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
# 85 "/usr/include/bits/mathcalls.h" 3 4
extern long double acoshl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __acoshl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double asinhl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __asinhl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double atanhl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __atanhl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern long double expl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __expl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double frexpl (long double __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__)); extern long double __frexpl (long double __x, int *__exponent) __attribute__ ((__nothrow__ , __leaf__));
extern long double ldexpl (long double __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__)); extern long double __ldexpl (long double __x, int __exponent) __attribute__ ((__nothrow__ , __leaf__));
 extern long double logl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __logl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double log10l (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __log10l (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double modfl (long double __x, long double *__iptr) __attribute__ ((__nothrow__ , __leaf__)); extern long double __modfl (long double __x, long double *__iptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 119 "/usr/include/bits/mathcalls.h" 3 4
extern long double expm1l (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __expm1l (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double log1pl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __log1pl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double logbl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __logbl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double exp2l (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __exp2l (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double log2l (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __log2l (long double __x) __attribute__ ((__nothrow__ , __leaf__));
 extern long double powl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __powl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double sqrtl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __sqrtl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double hypotl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __hypotl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double cbrtl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __cbrtl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double ceill (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __ceill (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double fabsl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __fabsl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double floorl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __floorl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double fmodl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __fmodl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
# 177 "/usr/include/bits/mathcalls.h" 3 4
extern int isinfl (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int finitel (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double dreml (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __dreml (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double significandl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __significandl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double copysignl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __copysignl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double nanl (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__)); extern long double __nanl (const char *__tagb) __attribute__ ((__nothrow__ , __leaf__));
# 211 "/usr/include/bits/mathcalls.h" 3 4
extern int isnanl (long double __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double j0l (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __j0l (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double j1l (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __j1l (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double jnl (int, long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __jnl (int, long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double y0l (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __y0l (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double y1l (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __y1l (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double ynl (int, long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __ynl (int, long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double erfl (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __erfl (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double erfcl (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __erfcl (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double lgammal (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __lgammal (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double tgammal (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __tgammal (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double gammal (long double) __attribute__ ((__nothrow__ , __leaf__)); extern long double __gammal (long double) __attribute__ ((__nothrow__ , __leaf__));
extern long double lgammal_r (long double, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__)); extern long double __lgammal_r (long double, int *__signgamp) __attribute__ ((__nothrow__ , __leaf__));
extern long double rintl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __rintl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double nextafterl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __nextafterl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double nexttowardl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __nexttowardl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/bits/mathcalls.h" 3 4
extern long double remainderl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __remainderl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double scalbnl (long double __x, int __n) __attribute__ ((__nothrow__ , __leaf__)); extern long double __scalbnl (long double __x, int __n) __attribute__ ((__nothrow__ , __leaf__));
extern int ilogbl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern int __ilogbl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
# 290 "/usr/include/bits/mathcalls.h" 3 4
extern long double scalblnl (long double __x, long int __n) __attribute__ ((__nothrow__ , __leaf__)); extern long double __scalblnl (long double __x, long int __n) __attribute__ ((__nothrow__ , __leaf__));
extern long double nearbyintl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long double __nearbyintl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double roundl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __roundl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double truncl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __truncl (long double __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double remquol (long double __x, long double __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__)); extern long double __remquol (long double __x, long double __y, int *__quo) __attribute__ ((__nothrow__ , __leaf__));
extern long int lrintl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lrintl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llrintl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llrintl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long int lroundl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long int __lroundl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
__extension__
extern long long int llroundl (long double __x) __attribute__ ((__nothrow__ , __leaf__)); extern long long int __llroundl (long double __x) __attribute__ ((__nothrow__ , __leaf__));
extern long double fdiml (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)); extern long double __fdiml (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__));
extern long double fmaxl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __fmaxl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double fminl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)); extern long double __fminl (long double __x, long double __y) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern long double fmal (long double __x, long double __y, long double __z) __attribute__ ((__nothrow__ , __leaf__)); extern long double __fmal (long double __x, long double __y, long double __z) __attribute__ ((__nothrow__ , __leaf__));
# 396 "/usr/include/bits/mathcalls.h" 3 4
extern long double scalbl (long double __x, long double __n) __attribute__ ((__nothrow__ , __leaf__)); extern long double __scalbl (long double __x, long double __n) __attribute__ ((__nothrow__ , __leaf__));
# 351 "/usr/include/math.h" 2 3 4
# 420 "/usr/include/math.h" 3 4
# 1 "/usr/include/bits/mathcalls-helper-functions.h" 1 3 4
# 21 "/usr/include/bits/mathcalls-helper-functions.h" 3 4
extern int __fpclassifyf128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __signbitf128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
extern int __isinff128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __finitef128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __isnanf128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int __iseqsigf128 (_Float128 __x, _Float128 __y) __attribute__ ((__nothrow__ , __leaf__));
extern int __issignalingf128 (_Float128 __value) __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__const__));
# 421 "/usr/include/math.h" 2 3 4
# 773 "/usr/include/math.h" 3 4
extern int signgam;
# 853 "/usr/include/math.h" 3 4
enum
  {
    FP_NAN =
      0,
    FP_INFINITE =
      1,
    FP_ZERO =
      2,
    FP_SUBNORMAL =
      3,
    FP_NORMAL =
      4
  };
# 1553 "/usr/include/math.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_MATH_H" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <math.h>

int main() {
double pi = M_PI;

if (pi);
;
  return 0;
}
            Found math #defines, like M_PI
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <math.h>

int main() {
double f = INFINITY;

if (f);
;
  return 0;
}
              Defined "HAVE_MATH_INFINITY" to "1"
            Found math INFINITY
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/socket.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/socket.h" 1 3 4
# 22 "/usr/include/sys/socket.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/types/struct_iovec.h" 1 3 4
# 23 "/usr/include/bits/types/struct_iovec.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 24 "/usr/include/bits/types/struct_iovec.h" 2 3 4
struct iovec
  {
    void *iov_base;
    size_t iov_len;
  };
# 27 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 29 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/socket.h" 1 3 4
# 27 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 28 "/usr/include/bits/socket.h" 2 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 30 "/usr/include/bits/socket.h" 2 3 4
typedef __socklen_t socklen_t;
# 1 "/usr/include/bits/socket_type.h" 1 3 4
# 24 "/usr/include/bits/socket_type.h" 3 4
enum __socket_type
{
  SOCK_STREAM = 1,
  SOCK_DGRAM = 2,
  SOCK_RAW = 3,
  SOCK_RDM = 4,
  SOCK_SEQPACKET = 5,
  SOCK_DCCP = 6,
  SOCK_PACKET = 10,
  SOCK_CLOEXEC = 02000000,
  SOCK_NONBLOCK = 00004000
};
# 39 "/usr/include/bits/socket.h" 2 3 4
# 172 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/bits/sockaddr.h" 1 3 4
# 28 "/usr/include/bits/sockaddr.h" 3 4
typedef unsigned short int sa_family_t;
# 173 "/usr/include/bits/socket.h" 2 3 4
struct sockaddr
  {
    sa_family_t sa_family;
    char sa_data[14];
  };
# 188 "/usr/include/bits/socket.h" 3 4
struct sockaddr_storage
  {
    sa_family_t ss_family;
    char __ss_padding[(128 - (sizeof (unsigned short int)) - sizeof (unsigned long int))];
    unsigned long int __ss_align;
  };
enum
  {
    MSG_OOB = 0x01,
    MSG_PEEK = 0x02,
    MSG_DONTROUTE = 0x04,
    MSG_CTRUNC = 0x08,
    MSG_PROXY = 0x10,
    MSG_TRUNC = 0x20,
    MSG_DONTWAIT = 0x40,
    MSG_EOR = 0x80,
    MSG_WAITALL = 0x100,
    MSG_FIN = 0x200,
    MSG_SYN = 0x400,
    MSG_CONFIRM = 0x800,
    MSG_RST = 0x1000,
    MSG_ERRQUEUE = 0x2000,
    MSG_NOSIGNAL = 0x4000,
    MSG_MORE = 0x8000,
    MSG_WAITFORONE = 0x10000,
    MSG_BATCH = 0x40000,
    MSG_ZEROCOPY = 0x4000000,
    MSG_FASTOPEN = 0x20000000,
    MSG_CMSG_CLOEXEC = 0x40000000
  };
struct msghdr
  {
    void *msg_name;
    socklen_t msg_namelen;
    struct iovec *msg_iov;
    size_t msg_iovlen;
    void *msg_control;
    size_t msg_controllen;
    int msg_flags;
  };
struct cmsghdr
  {
    size_t cmsg_len;
    int cmsg_level;
    int cmsg_type;
    __extension__ unsigned char __cmsg_data [];
  };
# 302 "/usr/include/bits/socket.h" 3 4
extern struct cmsghdr *__cmsg_nxthdr (struct msghdr *__mhdr,
          struct cmsghdr *__cmsg) __attribute__ ((__nothrow__ , __leaf__));
# 329 "/usr/include/bits/socket.h" 3 4
enum
  {
    SCM_RIGHTS = 0x01
  };
# 390 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/asm/socket.h" 1 3 4
# 1 "/usr/include/asm-generic/socket.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 1 3 4
# 1 "/usr/include/asm-generic/sockios.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 2 3 4
# 6 "/usr/include/asm-generic/socket.h" 2 3 4
# 1 "/usr/include/asm/socket.h" 2 3 4
# 391 "/usr/include/bits/socket.h" 2 3 4
# 444 "/usr/include/bits/socket.h" 3 4
struct linger
  {
    int l_onoff;
    int l_linger;
  };
# 34 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/types/struct_osockaddr.h" 1 3 4
struct osockaddr
{
  unsigned short int sa_family;
  unsigned char sa_data[14];
};
# 37 "/usr/include/sys/socket.h" 2 3 4
enum
{
  SHUT_RD = 0,
  SHUT_WR,
  SHUT_RDWR
};
# 102 "/usr/include/sys/socket.h" 3 4
extern int socket (int __domain, int __type, int __protocol) __attribute__ ((__nothrow__ , __leaf__));
extern int socketpair (int __domain, int __type, int __protocol,
         int __fds[2]) __attribute__ ((__nothrow__ , __leaf__));
extern int bind (int __fd, const struct sockaddr * __addr, socklen_t __len)
     __attribute__ ((__nothrow__ , __leaf__));
extern int getsockname (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
# 126 "/usr/include/sys/socket.h" 3 4
extern int connect (int __fd, const struct sockaddr * __addr, socklen_t __len);
extern int getpeername (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
extern ssize_t send (int __fd, const void *__buf, size_t __n, int __flags);
extern ssize_t recv (int __fd, void *__buf, size_t __n, int __flags);
extern ssize_t sendto (int __fd, const void *__buf, size_t __n,
         int __flags, const struct sockaddr * __addr,
         socklen_t __addr_len);
# 163 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvfrom (int __fd, void *__restrict __buf, size_t __n,
    int __flags, struct sockaddr *__restrict __addr,
    socklen_t *__restrict __addr_len);
extern ssize_t sendmsg (int __fd, const struct msghdr *__message,
   int __flags);
# 191 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvmsg (int __fd, struct msghdr *__message, int __flags);
# 208 "/usr/include/sys/socket.h" 3 4
extern int getsockopt (int __fd, int __level, int __optname,
         void *__restrict __optval,
         socklen_t *__restrict __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int setsockopt (int __fd, int __level, int __optname,
         const void *__optval, socklen_t __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int listen (int __fd, int __n) __attribute__ ((__nothrow__ , __leaf__));
# 232 "/usr/include/sys/socket.h" 3 4
extern int accept (int __fd, struct sockaddr *__restrict __addr,
     socklen_t *__restrict __addr_len);
# 250 "/usr/include/sys/socket.h" 3 4
extern int shutdown (int __fd, int __how) __attribute__ ((__nothrow__ , __leaf__));
extern int sockatmark (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int isfdtype (int __fd, int __fdtype) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/sys/socket.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_SOCKET_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/types.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/types.h" 1 3 4
# 25 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_TYPES_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: malloc.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/malloc.h" 1 3 4
# 22 "/usr/include/malloc.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/malloc.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long int ptrdiff_t;
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef struct {
  long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
  long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
# 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
} max_align_t;
# 24 "/usr/include/malloc.h" 2 3 4
# 1 "/usr/include/stdio.h" 1 3 4
# 27 "/usr/include/stdio.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 28 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 34 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 1 3 4
# 40 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 3 4
typedef __builtin_va_list __gnuc_va_list;
# 37 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 39 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types/__fpos_t.h" 1 3 4
# 1 "/usr/include/bits/types/__mbstate_t.h" 1 3 4
# 13 "/usr/include/bits/types/__mbstate_t.h" 3 4
typedef struct
{
  int __count;
  union
  {
    unsigned int __wch;
    char __wchb[4];
  } __value;
} __mbstate_t;
# 6 "/usr/include/bits/types/__fpos_t.h" 2 3 4
typedef struct _G_fpos_t
{
  __off_t __pos;
  __mbstate_t __state;
} __fpos_t;
# 40 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types/__fpos64_t.h" 1 3 4
# 10 "/usr/include/bits/types/__fpos64_t.h" 3 4
typedef struct _G_fpos64_t
{
  __off64_t __pos;
  __mbstate_t __state;
} __fpos64_t;
# 41 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types/__FILE.h" 1 3 4
struct _IO_FILE;
typedef struct _IO_FILE __FILE;
# 42 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types/FILE.h" 1 3 4
struct _IO_FILE;
typedef struct _IO_FILE FILE;
# 43 "/usr/include/stdio.h" 2 3 4
# 1 "/usr/include/bits/types/struct_FILE.h" 1 3 4
# 35 "/usr/include/bits/types/struct_FILE.h" 3 4
struct _IO_FILE;
struct _IO_marker;
struct _IO_codecvt;
struct _IO_wide_data;
typedef void _IO_lock_t;
struct _IO_FILE
{
  int _flags;
  char *_IO_read_ptr;
  char *_IO_read_end;
  char *_IO_read_base;
  char *_IO_write_base;
  char *_IO_write_ptr;
  char *_IO_write_end;
  char *_IO_buf_base;
  char *_IO_buf_end;
  char *_IO_save_base;
  char *_IO_backup_base;
  char *_IO_save_end;
  struct _IO_marker *_markers;
  struct _IO_FILE *_chain;
  int _fileno;
  int _flags2;
  __off_t _old_offset;
  unsigned short _cur_column;
  signed char _vtable_offset;
  char _shortbuf[1];
  _IO_lock_t *_lock;
  __off64_t _offset;
  struct _IO_codecvt *_codecvt;
  struct _IO_wide_data *_wide_data;
  struct _IO_FILE *_freeres_list;
  void *_freeres_buf;
  size_t __pad5;
  int _mode;
  char _unused2[15 * sizeof (int) - 4 * sizeof (void *) - sizeof (size_t)];
};
# 44 "/usr/include/stdio.h" 2 3 4
# 52 "/usr/include/stdio.h" 3 4
typedef __gnuc_va_list va_list;
# 63 "/usr/include/stdio.h" 3 4
typedef __off_t off_t;
# 77 "/usr/include/stdio.h" 3 4
typedef __ssize_t ssize_t;
typedef __fpos_t fpos_t;
# 133 "/usr/include/stdio.h" 3 4
# 1 "/usr/include/bits/stdio_lim.h" 1 3 4
# 134 "/usr/include/stdio.h" 2 3 4
extern FILE *stdin;
extern FILE *stdout;
extern FILE *stderr;
extern int remove (const char *__filename) __attribute__ ((__nothrow__ , __leaf__));
extern int rename (const char *__old, const char *__new) __attribute__ ((__nothrow__ , __leaf__));
extern int renameat (int __oldfd, const char *__old, int __newfd,
       const char *__new) __attribute__ ((__nothrow__ , __leaf__));
# 173 "/usr/include/stdio.h" 3 4
extern FILE *tmpfile (void) ;
# 187 "/usr/include/stdio.h" 3 4
extern char *tmpnam (char *__s) __attribute__ ((__nothrow__ , __leaf__)) ;
extern char *tmpnam_r (char *__s) __attribute__ ((__nothrow__ , __leaf__)) ;
# 204 "/usr/include/stdio.h" 3 4
extern char *tempnam (const char *__dir, const char *__pfx)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern int fclose (FILE *__stream);
extern int fflush (FILE *__stream);
# 227 "/usr/include/stdio.h" 3 4
extern int fflush_unlocked (FILE *__stream);
# 246 "/usr/include/stdio.h" 3 4
extern FILE *fopen (const char *__restrict __filename,
      const char *__restrict __modes) ;
extern FILE *freopen (const char *__restrict __filename,
        const char *__restrict __modes,
        FILE *__restrict __stream) ;
# 279 "/usr/include/stdio.h" 3 4
extern FILE *fdopen (int __fd, const char *__modes) __attribute__ ((__nothrow__ , __leaf__)) ;
# 292 "/usr/include/stdio.h" 3 4
extern FILE *fmemopen (void *__s, size_t __len, const char *__modes)
  __attribute__ ((__nothrow__ , __leaf__)) ;
extern FILE *open_memstream (char **__bufloc, size_t *__sizeloc) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void setbuf (FILE *__restrict __stream, char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern int setvbuf (FILE *__restrict __stream, char *__restrict __buf,
      int __modes, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern void setbuffer (FILE *__restrict __stream, char *__restrict __buf,
         size_t __size) __attribute__ ((__nothrow__ , __leaf__));
extern void setlinebuf (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
extern int fprintf (FILE *__restrict __stream,
      const char *__restrict __format, ...);
extern int printf (const char *__restrict __format, ...);
extern int sprintf (char *__restrict __s,
      const char *__restrict __format, ...) __attribute__ ((__nothrow__));
extern int vfprintf (FILE *__restrict __s, const char *__restrict __format,
       __gnuc_va_list __arg);
extern int vprintf (const char *__restrict __format, __gnuc_va_list __arg);
extern int vsprintf (char *__restrict __s, const char *__restrict __format,
       __gnuc_va_list __arg) __attribute__ ((__nothrow__));
extern int snprintf (char *__restrict __s, size_t __maxlen,
       const char *__restrict __format, ...)
     __attribute__ ((__nothrow__)) __attribute__ ((__format__ (__printf__, 3, 4)));
extern int vsnprintf (char *__restrict __s, size_t __maxlen,
        const char *__restrict __format, __gnuc_va_list __arg)
     __attribute__ ((__nothrow__)) __attribute__ ((__format__ (__printf__, 3, 0)));
# 379 "/usr/include/stdio.h" 3 4
extern int vdprintf (int __fd, const char *__restrict __fmt,
       __gnuc_va_list __arg)
     __attribute__ ((__format__ (__printf__, 2, 0)));
extern int dprintf (int __fd, const char *__restrict __fmt, ...)
     __attribute__ ((__format__ (__printf__, 2, 3)));
extern int fscanf (FILE *__restrict __stream,
     const char *__restrict __format, ...) ;
extern int scanf (const char *__restrict __format, ...) ;
extern int sscanf (const char *__restrict __s,
     const char *__restrict __format, ...) __attribute__ ((__nothrow__ , __leaf__));
# 409 "/usr/include/stdio.h" 3 4
extern int fscanf (FILE *__restrict __stream, const char *__restrict __format, ...) __asm__ ("" "__isoc99_fscanf")
                               ;
extern int scanf (const char *__restrict __format, ...) __asm__ ("" "__isoc99_scanf")
                              ;
extern int sscanf (const char *__restrict __s, const char *__restrict __format, ...) __asm__ ("" "__isoc99_sscanf") __attribute__ ((__nothrow__ , __leaf__))
                      ;
# 434 "/usr/include/stdio.h" 3 4
extern int vfscanf (FILE *__restrict __s, const char *__restrict __format,
      __gnuc_va_list __arg)
     __attribute__ ((__format__ (__scanf__, 2, 0))) ;
extern int vscanf (const char *__restrict __format, __gnuc_va_list __arg)
     __attribute__ ((__format__ (__scanf__, 1, 0))) ;
extern int vsscanf (const char *__restrict __s,
      const char *__restrict __format, __gnuc_va_list __arg)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__format__ (__scanf__, 2, 0)));
# 457 "/usr/include/stdio.h" 3 4
extern int vfscanf (FILE *__restrict __s, const char *__restrict __format, __gnuc_va_list __arg) __asm__ ("" "__isoc99_vfscanf")
     __attribute__ ((__format__ (__scanf__, 2, 0))) ;
extern int vscanf (const char *__restrict __format, __gnuc_va_list __arg) __asm__ ("" "__isoc99_vscanf")
     __attribute__ ((__format__ (__scanf__, 1, 0))) ;
extern int vsscanf (const char *__restrict __s, const char *__restrict __format, __gnuc_va_list __arg) __asm__ ("" "__isoc99_vsscanf") __attribute__ ((__nothrow__ , __leaf__))
     __attribute__ ((__format__ (__scanf__, 2, 0)));
# 491 "/usr/include/stdio.h" 3 4
extern int fgetc (FILE *__stream);
extern int getc (FILE *__stream);
extern int getchar (void);
extern int getc_unlocked (FILE *__stream);
extern int getchar_unlocked (void);
# 516 "/usr/include/stdio.h" 3 4
extern int fgetc_unlocked (FILE *__stream);
# 527 "/usr/include/stdio.h" 3 4
extern int fputc (int __c, FILE *__stream);
extern int putc (int __c, FILE *__stream);
extern int putchar (int __c);
# 543 "/usr/include/stdio.h" 3 4
extern int fputc_unlocked (int __c, FILE *__stream);
extern int putc_unlocked (int __c, FILE *__stream);
extern int putchar_unlocked (int __c);
extern int getw (FILE *__stream);
extern int putw (int __w, FILE *__stream);
extern char *fgets (char *__restrict __s, int __n, FILE *__restrict __stream)
     ;
# 609 "/usr/include/stdio.h" 3 4
extern __ssize_t __getdelim (char **__restrict __lineptr,
                             size_t *__restrict __n, int __delimiter,
                             FILE *__restrict __stream) ;
extern __ssize_t getdelim (char **__restrict __lineptr,
                           size_t *__restrict __n, int __delimiter,
                           FILE *__restrict __stream) ;
extern __ssize_t getline (char **__restrict __lineptr,
                          size_t *__restrict __n,
                          FILE *__restrict __stream) ;
extern int fputs (const char *__restrict __s, FILE *__restrict __stream);
extern int puts (const char *__s);
extern int ungetc (int __c, FILE *__stream);
extern size_t fread (void *__restrict __ptr, size_t __size,
       size_t __n, FILE *__restrict __stream) ;
extern size_t fwrite (const void *__restrict __ptr, size_t __size,
        size_t __n, FILE *__restrict __s);
# 679 "/usr/include/stdio.h" 3 4
extern size_t fread_unlocked (void *__restrict __ptr, size_t __size,
         size_t __n, FILE *__restrict __stream) ;
extern size_t fwrite_unlocked (const void *__restrict __ptr, size_t __size,
          size_t __n, FILE *__restrict __stream);
extern int fseek (FILE *__stream, long int __off, int __whence);
extern long int ftell (FILE *__stream) ;
extern void rewind (FILE *__stream);
# 713 "/usr/include/stdio.h" 3 4
extern int fseeko (FILE *__stream, __off_t __off, int __whence);
extern __off_t ftello (FILE *__stream) ;
# 737 "/usr/include/stdio.h" 3 4
extern int fgetpos (FILE *__restrict __stream, fpos_t *__restrict __pos);
extern int fsetpos (FILE *__stream, const fpos_t *__pos);
# 763 "/usr/include/stdio.h" 3 4
extern void clearerr (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
extern int feof (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int ferror (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void clearerr_unlocked (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
extern int feof_unlocked (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int ferror_unlocked (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void perror (const char *__s);
# 1 "/usr/include/bits/sys_errlist.h" 1 3 4
# 26 "/usr/include/bits/sys_errlist.h" 3 4
extern int sys_nerr;
extern const char *const sys_errlist[];
# 788 "/usr/include/stdio.h" 2 3 4
extern int fileno (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int fileno_unlocked (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
# 806 "/usr/include/stdio.h" 3 4
extern FILE *popen (const char *__command, const char *__modes) ;
extern int pclose (FILE *__stream);
extern char *ctermid (char *__s) __attribute__ ((__nothrow__ , __leaf__));
# 846 "/usr/include/stdio.h" 3 4
extern void flockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
extern int ftrylockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void funlockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
# 864 "/usr/include/stdio.h" 3 4
extern int __uflow (FILE *);
extern int __overflow (FILE *, int);
# 879 "/usr/include/stdio.h" 3 4
# 25 "/usr/include/malloc.h" 2 3 4
# 35 "/usr/include/malloc.h" 3 4
extern void *malloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
__attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
__attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
__attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
extern void *memalign (size_t __alignment, size_t __size)
__attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *valloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *pvalloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *(*__morecore) (ptrdiff_t __size);
extern void *__default_morecore (ptrdiff_t __size)
__attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__));
struct mallinfo
{
  int arena;
  int ordblks;
  int smblks;
  int hblks;
  int hblkhd;
  int usmblks;
  int fsmblks;
  int uordblks;
  int fordblks;
  int keepcost;
};
extern struct mallinfo mallinfo (void) __attribute__ ((__nothrow__ , __leaf__));
# 126 "/usr/include/malloc.h" 3 4
extern int mallopt (int __param, int __val) __attribute__ ((__nothrow__ , __leaf__));
extern int malloc_trim (size_t __pad) __attribute__ ((__nothrow__ , __leaf__));
extern size_t malloc_usable_size (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
extern void malloc_stats (void) __attribute__ ((__nothrow__ , __leaf__));
extern int malloc_info (int __options, FILE *__fp) __attribute__ ((__nothrow__ , __leaf__));
extern void (*volatile __free_hook) (void *__ptr,
                                                   const void *)
__attribute__ ((__deprecated__));
extern void *(*volatile __malloc_hook)(size_t __size,
                                                     const void *)
__attribute__ ((__deprecated__));
extern void *(*volatile __realloc_hook)(void *__ptr,
                                                      size_t __size,
                                                      const void *)
__attribute__ ((__deprecated__));
extern void *(*volatile __memalign_hook)(size_t __alignment,
                                                       size_t __size,
                                                       const void *)
__attribute__ ((__deprecated__));
extern void (*volatile __after_morecore_hook) (void);
extern void __malloc_check_init (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_MALLOC_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: time.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/time.h" 1 3 4
# 25 "/usr/include/time.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/time.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 30 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/time.h" 1 3 4
# 26 "/usr/include/bits/time.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 27 "/usr/include/bits/time.h" 2 3 4
# 34 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 38 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 39 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_tm.h" 1 3 4
struct tm
{
  int tm_sec;
  int tm_min;
  int tm_hour;
  int tm_mday;
  int tm_mon;
  int tm_year;
  int tm_wday;
  int tm_yday;
  int tm_isdst;
  long int tm_gmtoff;
  const char *tm_zone;
};
# 40 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 43 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 47 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 48 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_itimerspec.h" 1 3 4
struct itimerspec
  {
    struct timespec it_interval;
    struct timespec it_value;
  };
# 49 "/usr/include/time.h" 2 3 4
struct sigevent;
typedef __pid_t pid_t;
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 61 "/usr/include/time.h" 2 3 4
extern clock_t clock (void) __attribute__ ((__nothrow__ , __leaf__));
extern time_t time (time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern double difftime (time_t __time1, time_t __time0)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern time_t mktime (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern size_t strftime (char *__restrict __s, size_t __maxsize,
   const char *__restrict __format,
   const struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
# 104 "/usr/include/time.h" 3 4
extern size_t strftime_l (char *__restrict __s, size_t __maxsize,
     const char *__restrict __format,
     const struct tm *__restrict __tp,
     locale_t __loc) __attribute__ ((__nothrow__ , __leaf__));
# 119 "/usr/include/time.h" 3 4
extern struct tm *gmtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *gmtime_r (const time_t *__restrict __timer,
       struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime_r (const time_t *__restrict __timer,
          struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime (const struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime_r (const struct tm *__restrict __tp,
   char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime_r (const time_t *__restrict __timer,
        char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *__tzname[2];
extern int __daylight;
extern long int __timezone;
extern char *tzname[2];
extern void tzset (void) __attribute__ ((__nothrow__ , __leaf__));
extern int daylight;
extern long int timezone;
extern int stime (const time_t *__when) __attribute__ ((__nothrow__ , __leaf__));
# 196 "/usr/include/time.h" 3 4
extern time_t timegm (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern time_t timelocal (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int dysize (int __year) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 211 "/usr/include/time.h" 3 4
extern int nanosleep (const struct timespec *__requested_time,
        struct timespec *__remaining);
extern int clock_getres (clockid_t __clock_id, struct timespec *__res) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_gettime (clockid_t __clock_id, struct timespec *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_settime (clockid_t __clock_id, const struct timespec *__tp)
     __attribute__ ((__nothrow__ , __leaf__));
extern int clock_nanosleep (clockid_t __clock_id, int __flags,
       const struct timespec *__req,
       struct timespec *__rem);
extern int clock_getcpuclockid (pid_t __pid, clockid_t *__clock_id) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_create (clockid_t __clock_id,
    struct sigevent *__restrict __evp,
    timer_t *__restrict __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_delete (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_settime (timer_t __timerid, int __flags,
     const struct itimerspec *__restrict __value,
     struct itimerspec *__restrict __ovalue) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_gettime (timer_t __timerid, struct itimerspec *__value)
     __attribute__ ((__nothrow__ , __leaf__));
extern int timer_getoverrun (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timespec_get (struct timespec *__ts, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 307 "/usr/include/time.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_TIME_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: Direct.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Direct.h: No such file or directory
 #include <Direct.h>
          ^~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <Direct.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Direct.h: No such file or directory
 #include <Direct.h>
          ^~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Direct.h: No such file or directory #include <Direct.h>          ^~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: Ws2tcpip.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Ws2tcpip.h: No such file or directory
 #include <Ws2tcpip.h>
          ^~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <Ws2tcpip.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Ws2tcpip.h: No such file or directory
 #include <Ws2tcpip.h>
          ^~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: Ws2tcpip.h: No such file or directory #include <Ws2tcpip.h>          ^~~~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: endian.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/endian.h" 1 3 4
# 21 "/usr/include/endian.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/endian.h" 2 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 27 "/usr/include/bits/byteswap.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/bits/byteswap.h" 2 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_ENDIAN_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: immintrin.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 1 3 4
# 27 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mmintrin.h" 1 3 4
# 42 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mmintrin.h" 3 4
# 42 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mmintrin.h" 3 4
typedef int __m64 __attribute__ ((__vector_size__ (8), __may_alias__));
typedef int __m64_u __attribute__ ((__vector_size__ (8), __may_alias__, __aligned__ (1)));
typedef int __v2si __attribute__ ((__vector_size__ (8)));
typedef short __v4hi __attribute__ ((__vector_size__ (8)));
typedef char __v8qi __attribute__ ((__vector_size__ (8)));
typedef long long __v1di __attribute__ ((__vector_size__ (8)));
typedef float __v2sf __attribute__ ((__vector_size__ (8)));
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_empty (void)
{
  __builtin_ia32_emms ();
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_empty (void)
{
  _mm_empty ();
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi32_si64 (int __i)
{
  return (__m64) __builtin_ia32_vec_init_v2si (__i, 0);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_from_int (int __i)
{
  return _mm_cvtsi32_si64 (__i);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_from_int64 (long long __i)
{
  return (__m64) __i;
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_m64 (long long __i)
{
  return (__m64) __i;
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64x_si64 (long long __i)
{
  return (__m64) __i;
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pi64x (long long __i)
{
  return (__m64) __i;
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_si32 (__m64 __i)
{
  return __builtin_ia32_vec_ext_v2si ((__v2si)__i, 0);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_to_int (__m64 __i)
{
  return _mm_cvtsi64_si32 (__i);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_to_int64 (__m64 __i)
{
  return (long long)__i;
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtm64_si64 (__m64 __i)
{
  return (long long)__i;
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_si64x (__m64 __i)
{
  return (long long)__i;
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packs_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_packsswb ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_packsswb (__m64 __m1, __m64 __m2)
{
  return _mm_packs_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packs_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_packssdw ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_packssdw (__m64 __m1, __m64 __m2)
{
  return _mm_packs_pi32 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packs_pu16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_packuswb ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_packuswb (__m64 __m1, __m64 __m2)
{
  return _mm_packs_pu16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpckhbw ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpckhbw (__m64 __m1, __m64 __m2)
{
  return _mm_unpackhi_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpckhwd ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpckhwd (__m64 __m1, __m64 __m2)
{
  return _mm_unpackhi_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpckhdq ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpckhdq (__m64 __m1, __m64 __m2)
{
  return _mm_unpackhi_pi32 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpcklbw ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpcklbw (__m64 __m1, __m64 __m2)
{
  return _mm_unpacklo_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpcklwd ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpcklwd (__m64 __m1, __m64 __m2)
{
  return _mm_unpacklo_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_punpckldq ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_punpckldq (__m64 __m1, __m64 __m2)
{
  return _mm_unpacklo_pi32 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddb (__m64 __m1, __m64 __m2)
{
  return _mm_add_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddw (__m64 __m1, __m64 __m2)
{
  return _mm_add_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddd ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddd (__m64 __m1, __m64 __m2)
{
  return _mm_add_pi32 (__m1, __m2);
}
# 322 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mmintrin.h" 3 4
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_si64 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddq ((__v1di)__m1, (__v1di)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddsb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddsb (__m64 __m1, __m64 __m2)
{
  return _mm_adds_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddsw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddsw (__m64 __m1, __m64 __m2)
{
  return _mm_adds_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_pu8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddusb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddusb (__m64 __m1, __m64 __m2)
{
  return _mm_adds_pu8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_pu16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_paddusw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_paddusw (__m64 __m1, __m64 __m2)
{
  return _mm_adds_pu16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubb (__m64 __m1, __m64 __m2)
{
  return _mm_sub_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubw (__m64 __m1, __m64 __m2)
{
  return _mm_sub_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubd ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubd (__m64 __m1, __m64 __m2)
{
  return _mm_sub_pi32 (__m1, __m2);
}
# 434 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mmintrin.h" 3 4
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_si64 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubq ((__v1di)__m1, (__v1di)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubsb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubsb (__m64 __m1, __m64 __m2)
{
  return _mm_subs_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubsw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubsw (__m64 __m1, __m64 __m2)
{
  return _mm_subs_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_pu8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubusb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubusb (__m64 __m1, __m64 __m2)
{
  return _mm_subs_pu8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_pu16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_psubusw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psubusw (__m64 __m1, __m64 __m2)
{
  return _mm_subs_pu16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_madd_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pmaddwd ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmaddwd (__m64 __m1, __m64 __m2)
{
  return _mm_madd_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhi_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pmulhw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmulhw (__m64 __m1, __m64 __m2)
{
  return _mm_mulhi_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mullo_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pmullw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmullw (__m64 __m1, __m64 __m2)
{
  return _mm_mullo_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_pi16 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psllw ((__v4hi)__m, (__v4hi)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psllw (__m64 __m, __m64 __count)
{
  return _mm_sll_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_pi16 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psllwi ((__v4hi)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psllwi (__m64 __m, int __count)
{
  return _mm_slli_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_pi32 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_pslld ((__v2si)__m, (__v2si)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pslld (__m64 __m, __m64 __count)
{
  return _mm_sll_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_pi32 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_pslldi ((__v2si)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pslldi (__m64 __m, int __count)
{
  return _mm_slli_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_si64 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psllq ((__v1di)__m, (__v1di)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psllq (__m64 __m, __m64 __count)
{
  return _mm_sll_si64 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_si64 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psllqi ((__v1di)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psllqi (__m64 __m, int __count)
{
  return _mm_slli_si64 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sra_pi16 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psraw ((__v4hi)__m, (__v4hi)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psraw (__m64 __m, __m64 __count)
{
  return _mm_sra_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srai_pi16 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psrawi ((__v4hi)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrawi (__m64 __m, int __count)
{
  return _mm_srai_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sra_pi32 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psrad ((__v2si)__m, (__v2si)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrad (__m64 __m, __m64 __count)
{
  return _mm_sra_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srai_pi32 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psradi ((__v2si)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psradi (__m64 __m, int __count)
{
  return _mm_srai_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_pi16 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psrlw ((__v4hi)__m, (__v4hi)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrlw (__m64 __m, __m64 __count)
{
  return _mm_srl_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_pi16 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psrlwi ((__v4hi)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrlwi (__m64 __m, int __count)
{
  return _mm_srli_pi16 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_pi32 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psrld ((__v2si)__m, (__v2si)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrld (__m64 __m, __m64 __count)
{
  return _mm_srl_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_pi32 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psrldi ((__v2si)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrldi (__m64 __m, int __count)
{
  return _mm_srli_pi32 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_si64 (__m64 __m, __m64 __count)
{
  return (__m64) __builtin_ia32_psrlq ((__v1di)__m, (__v1di)__count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrlq (__m64 __m, __m64 __count)
{
  return _mm_srl_si64 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_si64 (__m64 __m, int __count)
{
  return (__m64) __builtin_ia32_psrlqi ((__v1di)__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psrlqi (__m64 __m, int __count)
{
  return _mm_srli_si64 (__m, __count);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_and_si64 (__m64 __m1, __m64 __m2)
{
  return __builtin_ia32_pand (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pand (__m64 __m1, __m64 __m2)
{
  return _mm_and_si64 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_andnot_si64 (__m64 __m1, __m64 __m2)
{
  return __builtin_ia32_pandn (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pandn (__m64 __m1, __m64 __m2)
{
  return _mm_andnot_si64 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_or_si64 (__m64 __m1, __m64 __m2)
{
  return __builtin_ia32_por (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_por (__m64 __m1, __m64 __m2)
{
  return _mm_or_si64 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_xor_si64 (__m64 __m1, __m64 __m2)
{
  return __builtin_ia32_pxor (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pxor (__m64 __m1, __m64 __m2)
{
  return _mm_xor_si64 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpeqb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpeqb (__m64 __m1, __m64 __m2)
{
  return _mm_cmpeq_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_pi8 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpgtb ((__v8qi)__m1, (__v8qi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpgtb (__m64 __m1, __m64 __m2)
{
  return _mm_cmpgt_pi8 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpeqw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpeqw (__m64 __m1, __m64 __m2)
{
  return _mm_cmpeq_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_pi16 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpgtw ((__v4hi)__m1, (__v4hi)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpgtw (__m64 __m1, __m64 __m2)
{
  return _mm_cmpgt_pi16 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpeqd ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpeqd (__m64 __m1, __m64 __m2)
{
  return _mm_cmpeq_pi32 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_pi32 (__m64 __m1, __m64 __m2)
{
  return (__m64) __builtin_ia32_pcmpgtd ((__v2si)__m1, (__v2si)__m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pcmpgtd (__m64 __m1, __m64 __m2)
{
  return _mm_cmpgt_pi32 (__m1, __m2);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setzero_si64 (void)
{
  return (__m64)0LL;
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pi32 (int __i1, int __i0)
{
  return (__m64) __builtin_ia32_vec_init_v2si (__i0, __i1);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pi16 (short __w3, short __w2, short __w1, short __w0)
{
  return (__m64) __builtin_ia32_vec_init_v4hi (__w0, __w1, __w2, __w3);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pi8 (char __b7, char __b6, char __b5, char __b4,
      char __b3, char __b2, char __b1, char __b0)
{
  return (__m64) __builtin_ia32_vec_init_v8qi (__b0, __b1, __b2, __b3,
            __b4, __b5, __b6, __b7);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_pi32 (int __i0, int __i1)
{
  return _mm_set_pi32 (__i1, __i0);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_pi16 (short __w0, short __w1, short __w2, short __w3)
{
  return _mm_set_pi16 (__w3, __w2, __w1, __w0);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_pi8 (char __b0, char __b1, char __b2, char __b3,
       char __b4, char __b5, char __b6, char __b7)
{
  return _mm_set_pi8 (__b7, __b6, __b5, __b4, __b3, __b2, __b1, __b0);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_pi32 (int __i)
{
  return _mm_set_pi32 (__i, __i);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_pi16 (short __w)
{
  return _mm_set_pi16 (__w, __w, __w, __w);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_pi8 (char __b)
{
  return _mm_set_pi8 (__b, __b, __b, __b, __b, __b, __b, __b);
}
# 28 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 1 3 4
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mm_malloc.h" 1 3 4
# 27 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mm_malloc.h" 3 4
# 1 "/usr/include/stdlib.h" 1 3 4
# 25 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 26 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 32 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 40 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 41 "/usr/include/stdlib.h" 2 3 4
# 55 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 56 "/usr/include/stdlib.h" 2 3 4
typedef struct
  {
    int quot;
    int rem;
  } div_t;
typedef struct
  {
    long int quot;
    long int rem;
  } ldiv_t;
__extension__ typedef struct
  {
    long long int quot;
    long long int rem;
  } lldiv_t;
# 97 "/usr/include/stdlib.h" 3 4
extern size_t __ctype_get_mb_cur_max (void) __attribute__ ((__nothrow__ , __leaf__)) ;
extern double atof (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern int atoi (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern long int atol (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
__extension__ extern long long int atoll (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern double strtod (const char *__restrict __nptr,
        char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern float strtof (const char *__restrict __nptr,
       char **__restrict __endptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long double strtold (const char *__restrict __nptr,
       char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 176 "/usr/include/stdlib.h" 3 4
extern long int strtol (const char *__restrict __nptr,
   char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern unsigned long int strtoul (const char *__restrict __nptr,
      char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoq (const char *__restrict __nptr,
        char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtouq (const char *__restrict __nptr,
           char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoll (const char *__restrict __nptr,
         char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtoull (const char *__restrict __nptr,
     char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 385 "/usr/include/stdlib.h" 3 4
extern char *l64a (long int __n) __attribute__ ((__nothrow__ , __leaf__)) ;
extern long int a64l (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 395 "/usr/include/stdlib.h" 2 3 4
extern long int random (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srandom (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern char *initstate (unsigned int __seed, char *__statebuf,
   size_t __statelen) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *setstate (char *__statebuf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct random_data
  {
    int32_t *fptr;
    int32_t *rptr;
    int32_t *state;
    int rand_type;
    int rand_deg;
    int rand_sep;
    int32_t *end_ptr;
  };
extern int random_r (struct random_data *__restrict __buf,
       int32_t *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srandom_r (unsigned int __seed, struct random_data *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int initstate_r (unsigned int __seed, char *__restrict __statebuf,
   size_t __statelen,
   struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern int setstate_r (char *__restrict __statebuf,
         struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int rand (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srand (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern int rand_r (unsigned int *__seed) __attribute__ ((__nothrow__ , __leaf__));
extern double drand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern double erand48 (unsigned short int __xsubi[3]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int lrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int nrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int mrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int jrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void srand48 (long int __seedval) __attribute__ ((__nothrow__ , __leaf__));
extern unsigned short int *seed48 (unsigned short int __seed16v[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void lcong48 (unsigned short int __param[7]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct drand48_data
  {
    unsigned short int __x[3];
    unsigned short int __old_x[3];
    unsigned short int __c;
    unsigned short int __init;
    __extension__ unsigned long long int __a;
  };
extern int drand48_r (struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int erand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int nrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int mrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int jrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srand48_r (long int __seedval, struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int seed48_r (unsigned short int __seed16v[3],
       struct drand48_data *__buffer) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lcong48_r (unsigned short int __param[7],
        struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *malloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
# 563 "/usr/include/stdlib.h" 3 4
extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/alloca.h" 1 3 4
# 24 "/usr/include/alloca.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 25 "/usr/include/alloca.h" 2 3 4
extern void *alloca (size_t __size) __attribute__ ((__nothrow__ , __leaf__));
# 567 "/usr/include/stdlib.h" 2 3 4
extern void *valloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern void *aligned_alloc (size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__alloc_size__ (2))) ;
extern void abort (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern int atexit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int at_quick_exit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int on_exit (void (*__func) (int __status, void *__arg), void *__arg)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void quick_exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void _Exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern char *getenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 644 "/usr/include/stdlib.h" 3 4
extern int putenv (char *__string) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int setenv (const char *__name, const char *__value, int __replace)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int unsetenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int clearenv (void) __attribute__ ((__nothrow__ , __leaf__));
# 672 "/usr/include/stdlib.h" 3 4
extern char *mktemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 685 "/usr/include/stdlib.h" 3 4
extern int mkstemp (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 707 "/usr/include/stdlib.h" 3 4
extern int mkstemps (char *__template, int __suffixlen) __attribute__ ((__nonnull__ (1))) ;
# 728 "/usr/include/stdlib.h" 3 4
extern char *mkdtemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 781 "/usr/include/stdlib.h" 3 4
extern int system (const char *__command) ;
# 797 "/usr/include/stdlib.h" 3 4
extern char *realpath (const char *__restrict __name,
         char *__restrict __resolved) __attribute__ ((__nothrow__ , __leaf__)) ;
typedef int (*__compar_fn_t) (const void *, const void *);
# 817 "/usr/include/stdlib.h" 3 4
extern void *bsearch (const void *__key, const void *__base,
        size_t __nmemb, size_t __size, __compar_fn_t __compar)
     __attribute__ ((__nonnull__ (1, 2, 5))) ;
extern void qsort (void *__base, size_t __nmemb, size_t __size,
     __compar_fn_t __compar) __attribute__ ((__nonnull__ (1, 4)));
# 837 "/usr/include/stdlib.h" 3 4
extern int abs (int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern long int labs (long int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern long long int llabs (long long int __x)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern div_t div (int __numer, int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern ldiv_t ldiv (long int __numer, long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern lldiv_t lldiv (long long int __numer,
        long long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
# 869 "/usr/include/stdlib.h" 3 4
extern char *ecvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *fcvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *gcvt (double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern char *qecvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qfcvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qgcvt (long double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern int ecvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int fcvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qecvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qfcvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int mblen (const char *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int mbtowc (wchar_t *__restrict __pwc,
     const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int wctomb (char *__s, wchar_t __wchar) __attribute__ ((__nothrow__ , __leaf__));
extern size_t mbstowcs (wchar_t *__restrict __pwcs,
   const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern size_t wcstombs (char *__restrict __s,
   const wchar_t *__restrict __pwcs, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__));
extern int rpmatch (const char *__response) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 954 "/usr/include/stdlib.h" 3 4
extern int getsubopt (char **__restrict __optionp,
        char *const *__restrict __tokens,
        char **__restrict __valuep)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3))) ;
# 1000 "/usr/include/stdlib.h" 3 4
extern int getloadavg (double __loadavg[], int __nelem)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1010 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/stdlib-float.h" 1 3 4
# 1011 "/usr/include/stdlib.h" 2 3 4
# 1020 "/usr/include/stdlib.h" 3 4
# 28 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/mm_malloc.h" 2 3 4
extern int posix_memalign (void **, size_t, size_t);
static __inline void *
_mm_malloc (size_t __size, size_t __alignment)
{
  void *__ptr;
  if (__alignment == 1)
    return malloc (__size);
  if (__alignment == 2 || (sizeof (void *) == 8 && __alignment == 4))
    __alignment = sizeof (void *);
  if (posix_memalign (&__ptr, __alignment, __size) == 0)
    return __ptr;
  else
    return ((void *)0);
}
static __inline void
_mm_free (void *__ptr)
{
  free (__ptr);
}
# 35 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 2 3 4
enum _mm_hint
{
  _MM_HINT_ET0 = 7,
  _MM_HINT_ET1 = 6,
  _MM_HINT_T0 = 3,
  _MM_HINT_T1 = 2,
  _MM_HINT_T2 = 1,
  _MM_HINT_NTA = 0
};
# 69 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
typedef float __m128 __attribute__ ((__vector_size__ (16), __may_alias__));
typedef float __m128_u __attribute__ ((__vector_size__ (16), __may_alias__, __aligned__ (1)));
typedef float __v4sf __attribute__ ((__vector_size__ (16)));
# 109 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_undefined_ps (void)
{
  __m128 __Y = __Y;
  return __Y;
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setzero_ps (void)
{
  return __extension__ (__m128){ 0.0f, 0.0f, 0.0f, 0.0f };
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_addss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_subss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_mulss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_div_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_divss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sqrt_ss (__m128 __A)
{
  return (__m128) __builtin_ia32_sqrtss ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp_ss (__m128 __A)
{
  return (__m128) __builtin_ia32_rcpss ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt_ss (__m128 __A)
{
  return (__m128) __builtin_ia32_rsqrtss ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_ps (__m128 __A, __m128 __B)
{
  return (__m128) ((__v4sf)__A + (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_ps (__m128 __A, __m128 __B)
{
  return (__m128) ((__v4sf)__A - (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_ps (__m128 __A, __m128 __B)
{
  return (__m128) ((__v4sf)__A * (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_div_ps (__m128 __A, __m128 __B)
{
  return (__m128) ((__v4sf)__A / (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sqrt_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_sqrtps ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_rcpps ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_rsqrtps ((__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_and_ps (__m128 __A, __m128 __B)
{
  return __builtin_ia32_andps (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_andnot_ps (__m128 __A, __m128 __B)
{
  return __builtin_ia32_andnps (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_or_ps (__m128 __A, __m128 __B)
{
  return __builtin_ia32_orps (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_xor_ps (__m128 __A, __m128 __B)
{
  return __builtin_ia32_xorps (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpeqss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpltss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpless ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movss ((__v4sf) __A,
     (__v4sf)
     __builtin_ia32_cmpltss ((__v4sf) __B,
        (__v4sf)
        __A));
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movss ((__v4sf) __A,
     (__v4sf)
     __builtin_ia32_cmpless ((__v4sf) __B,
        (__v4sf)
        __A));
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpneqss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnlt_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpnltss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnle_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpnless ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpngt_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movss ((__v4sf) __A,
     (__v4sf)
     __builtin_ia32_cmpnltss ((__v4sf) __B,
         (__v4sf)
         __A));
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnge_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movss ((__v4sf) __A,
     (__v4sf)
     __builtin_ia32_cmpnless ((__v4sf) __B,
         (__v4sf)
         __A));
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpord_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpordss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpunord_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpunordss ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpeqps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpltps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpleps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpgtps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpgeps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpneqps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnlt_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpnltps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnle_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpnleps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpngt_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpngtps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnge_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpngeps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpord_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpordps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpunord_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_cmpunordps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comieq_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comieq ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comilt_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comilt ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comile_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comile ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comigt_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comigt ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comige_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comige ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comineq_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_comineq ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomieq_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomieq ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomilt_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomilt ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomile_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomile ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomigt_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomigt ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomige_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomige ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomineq_ss (__m128 __A, __m128 __B)
{
  return __builtin_ia32_ucomineq ((__v4sf)__A, (__v4sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_si32 (__m128 __A)
{
  return __builtin_ia32_cvtss2si ((__v4sf) __A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvt_ss2si (__m128 __A)
{
  return _mm_cvtss_si32 (__A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_si64 (__m128 __A)
{
  return __builtin_ia32_cvtss2si64 ((__v4sf) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_si64x (__m128 __A)
{
  return __builtin_ia32_cvtss2si64 ((__v4sf) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_pi32 (__m128 __A)
{
  return (__m64) __builtin_ia32_cvtps2pi ((__v4sf) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvt_ps2pi (__m128 __A)
{
  return _mm_cvtps_pi32 (__A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_si32 (__m128 __A)
{
  return __builtin_ia32_cvttss2si ((__v4sf) __A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtt_ss2si (__m128 __A)
{
  return _mm_cvttss_si32 (__A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_si64 (__m128 __A)
{
  return __builtin_ia32_cvttss2si64 ((__v4sf) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_si64x (__m128 __A)
{
  return __builtin_ia32_cvttss2si64 ((__v4sf) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttps_pi32 (__m128 __A)
{
  return (__m64) __builtin_ia32_cvttps2pi ((__v4sf) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtt_ps2pi (__m128 __A)
{
  return _mm_cvttps_pi32 (__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi32_ss (__m128 __A, int __B)
{
  return (__m128) __builtin_ia32_cvtsi2ss ((__v4sf) __A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvt_si2ss (__m128 __A, int __B)
{
  return _mm_cvtsi32_ss (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_ss (__m128 __A, long long __B)
{
  return (__m128) __builtin_ia32_cvtsi642ss ((__v4sf) __A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64x_ss (__m128 __A, long long __B)
{
  return (__m128) __builtin_ia32_cvtsi642ss ((__v4sf) __A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpi32_ps (__m128 __A, __m64 __B)
{
  return (__m128) __builtin_ia32_cvtpi2ps ((__v4sf) __A, (__v2si)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvt_pi2ps (__m128 __A, __m64 __B)
{
  return _mm_cvtpi32_ps (__A, __B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpi16_ps (__m64 __A)
{
  __v4hi __sign;
  __v2si __hisi, __losi;
  __v4sf __zero, __ra, __rb;
  __sign = __builtin_ia32_pcmpgtw ((__v4hi)0LL, (__v4hi)__A);
  __losi = (__v2si) __builtin_ia32_punpcklwd ((__v4hi)__A, __sign);
  __hisi = (__v2si) __builtin_ia32_punpckhwd ((__v4hi)__A, __sign);
  __zero = (__v4sf) _mm_setzero_ps ();
  __ra = __builtin_ia32_cvtpi2ps (__zero, __losi);
  __rb = __builtin_ia32_cvtpi2ps (__ra, __hisi);
  return (__m128) __builtin_ia32_movlhps (__ra, __rb);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpu16_ps (__m64 __A)
{
  __v2si __hisi, __losi;
  __v4sf __zero, __ra, __rb;
  __losi = (__v2si) __builtin_ia32_punpcklwd ((__v4hi)__A, (__v4hi)0LL);
  __hisi = (__v2si) __builtin_ia32_punpckhwd ((__v4hi)__A, (__v4hi)0LL);
  __zero = (__v4sf) _mm_setzero_ps ();
  __ra = __builtin_ia32_cvtpi2ps (__zero, __losi);
  __rb = __builtin_ia32_cvtpi2ps (__ra, __hisi);
  return (__m128) __builtin_ia32_movlhps (__ra, __rb);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpi8_ps (__m64 __A)
{
  __v8qi __sign;
  __sign = __builtin_ia32_pcmpgtb ((__v8qi)0LL, (__v8qi)__A);
  __A = (__m64) __builtin_ia32_punpcklbw ((__v8qi)__A, __sign);
  return _mm_cvtpi16_ps(__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpu8_ps(__m64 __A)
{
  __A = (__m64) __builtin_ia32_punpcklbw ((__v8qi)__A, (__v8qi)0LL);
  return _mm_cvtpu16_ps(__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpi32x2_ps(__m64 __A, __m64 __B)
{
  __v4sf __zero = (__v4sf) _mm_setzero_ps ();
  __v4sf __sfa = __builtin_ia32_cvtpi2ps (__zero, (__v2si)__A);
  __v4sf __sfb = __builtin_ia32_cvtpi2ps (__sfa, (__v2si)__B);
  return (__m128) __builtin_ia32_movlhps (__sfa, __sfb);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_pi16(__m128 __A)
{
  __v4sf __hisf = (__v4sf)__A;
  __v4sf __losf = __builtin_ia32_movhlps (__hisf, __hisf);
  __v2si __hisi = __builtin_ia32_cvtps2pi (__hisf);
  __v2si __losi = __builtin_ia32_cvtps2pi (__losf);
  return (__m64) __builtin_ia32_packssdw (__hisi, __losi);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_pi8(__m128 __A)
{
  __v4hi __tmp = (__v4hi) _mm_cvtps_pi16 (__A);
  return (__m64) __builtin_ia32_packsswb (__tmp, (__v4hi)0LL);
}
# 755 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpckhps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpcklps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadh_pi (__m128 __A, __m64 const *__P)
{
  return (__m128) __builtin_ia32_loadhps ((__v4sf)__A, (const __v2sf *)__P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storeh_pi (__m64 *__P, __m128 __A)
{
  __builtin_ia32_storehps ((__v2sf *)__P, (__v4sf)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movehl_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movhlps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movelh_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movlhps ((__v4sf)__A, (__v4sf)__B);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadl_pi (__m128 __A, __m64 const *__P)
{
  return (__m128) __builtin_ia32_loadlps ((__v4sf)__A, (const __v2sf *)__P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storel_pi (__m64 *__P, __m128 __A)
{
  __builtin_ia32_storelps ((__v2sf *)__P, (__v4sf)__A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movemask_ps (__m128 __A)
{
  return __builtin_ia32_movmskps ((__v4sf)__A);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_getcsr (void)
{
  return __builtin_ia32_stmxcsr ();
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_GET_EXCEPTION_STATE (void)
{
  return _mm_getcsr() & 0x003f;
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_GET_EXCEPTION_MASK (void)
{
  return _mm_getcsr() & 0x1f80;
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_GET_ROUNDING_MODE (void)
{
  return _mm_getcsr() & 0x6000;
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_GET_FLUSH_ZERO_MODE (void)
{
  return _mm_getcsr() & 0x8000;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setcsr (unsigned int __I)
{
  __builtin_ia32_ldmxcsr (__I);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_SET_EXCEPTION_STATE(unsigned int __mask)
{
  _mm_setcsr((_mm_getcsr() & ~0x003f) | __mask);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_SET_EXCEPTION_MASK (unsigned int __mask)
{
  _mm_setcsr((_mm_getcsr() & ~0x1f80) | __mask);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_SET_ROUNDING_MODE (unsigned int __mode)
{
  _mm_setcsr((_mm_getcsr() & ~0x6000) | __mode);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_MM_SET_FLUSH_ZERO_MODE (unsigned int __mode)
{
  _mm_setcsr((_mm_getcsr() & ~0x8000) | __mode);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_ss (float __F)
{
  return __extension__ (__m128)(__v4sf){ __F, 0.0f, 0.0f, 0.0f };
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_ps (float __F)
{
  return __extension__ (__m128)(__v4sf){ __F, __F, __F, __F };
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_ps1 (float __F)
{
  return _mm_set1_ps (__F);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_ss (float const *__P)
{
  return _mm_set_ss (*__P);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load1_ps (float const *__P)
{
  return _mm_set1_ps (*__P);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_ps1 (float const *__P)
{
  return _mm_load1_ps (__P);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_ps (float const *__P)
{
  return *(__m128 *)__P;
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadu_ps (float const *__P)
{
  return *(__m128_u *)__P;
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadr_ps (float const *__P)
{
  __v4sf __tmp = *(__v4sf *)__P;
  return (__m128) __builtin_ia32_shufps (__tmp, __tmp, (((0) << 6) | ((1) << 4) | ((2) << 2) | (3)));
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_ps (const float __Z, const float __Y, const float __X, const float __W)
{
  return __extension__ (__m128)(__v4sf){ __W, __X, __Y, __Z };
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_ps (float __Z, float __Y, float __X, float __W)
{
  return __extension__ (__m128)(__v4sf){ __Z, __Y, __X, __W };
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_ss (float *__P, __m128 __A)
{
  *__P = ((__v4sf)__A)[0];
}
extern __inline float __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_f32 (__m128 __A)
{
  return ((__v4sf)__A)[0];
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_ps (float *__P, __m128 __A)
{
  *(__m128 *)__P = __A;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storeu_ps (float *__P, __m128 __A)
{
  *(__m128_u *)__P = __A;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store1_ps (float *__P, __m128 __A)
{
  __v4sf __va = (__v4sf)__A;
  __v4sf __tmp = __builtin_ia32_shufps (__va, __va, (((0) << 6) | ((0) << 4) | ((0) << 2) | (0)));
  _mm_storeu_ps (__P, __tmp);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_ps1 (float *__P, __m128 __A)
{
  _mm_store1_ps (__P, __A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storer_ps (float *__P, __m128 __A)
{
  __v4sf __va = (__v4sf)__A;
  __v4sf __tmp = __builtin_ia32_shufps (__va, __va, (((0) << 6) | ((1) << 4) | ((2) << 2) | (3)));
  _mm_store_ps (__P, __tmp);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_move_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_movss ((__v4sf)__A, (__v4sf)__B);
}
# 1060 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_pi16 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pmaxsw ((__v4hi)__A, (__v4hi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmaxsw (__m64 __A, __m64 __B)
{
  return _mm_max_pi16 (__A, __B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_pu8 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pmaxub ((__v8qi)__A, (__v8qi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmaxub (__m64 __A, __m64 __B)
{
  return _mm_max_pu8 (__A, __B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_pi16 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pminsw ((__v4hi)__A, (__v4hi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pminsw (__m64 __A, __m64 __B)
{
  return _mm_min_pi16 (__A, __B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_pu8 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pminub ((__v8qi)__A, (__v8qi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pminub (__m64 __A, __m64 __B)
{
  return _mm_min_pu8 (__A, __B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movemask_pi8 (__m64 __A)
{
  return __builtin_ia32_pmovmskb ((__v8qi)__A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmovmskb (__m64 __A)
{
  return _mm_movemask_pi8 (__A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhi_pu16 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pmulhuw ((__v4hi)__A, (__v4hi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pmulhuw (__m64 __A, __m64 __B)
{
  return _mm_mulhi_pu16 (__A, __B);
}
# 1162 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskmove_si64 (__m64 __A, __m64 __N, char *__P)
{
  __builtin_ia32_maskmovq ((__v8qi)__A, (__v8qi)__N, __P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_maskmovq (__m64 __A, __m64 __N, char *__P)
{
  _mm_maskmove_si64 (__A, __N, __P);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_avg_pu8 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pavgb ((__v8qi)__A, (__v8qi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pavgb (__m64 __A, __m64 __B)
{
  return _mm_avg_pu8 (__A, __B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_avg_pu16 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_pavgw ((__v4hi)__A, (__v4hi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_pavgw (__m64 __A, __m64 __B)
{
  return _mm_avg_pu16 (__A, __B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sad_pu8 (__m64 __A, __m64 __B)
{
  return (__m64) __builtin_ia32_psadbw ((__v8qi)__A, (__v8qi)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_m_psadbw (__m64 __A, __m64 __B)
{
  return _mm_sad_pu8 (__A, __B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_pi (__m64 *__P, __m64 __A)
{
  __builtin_ia32_movntq ((unsigned long long *)__P, (unsigned long long)__A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_ps (float *__P, __m128 __A)
{
  __builtin_ia32_movntps (__P, (__v4sf)__A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sfence (void)
{
  __builtin_ia32_sfence ();
}
# 1252 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 1 3 4
# 31 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 2 3 4
# 40 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
typedef double __v2df __attribute__ ((__vector_size__ (16)));
typedef long long __v2di __attribute__ ((__vector_size__ (16)));
typedef unsigned long long __v2du __attribute__ ((__vector_size__ (16)));
typedef int __v4si __attribute__ ((__vector_size__ (16)));
typedef unsigned int __v4su __attribute__ ((__vector_size__ (16)));
typedef short __v8hi __attribute__ ((__vector_size__ (16)));
typedef unsigned short __v8hu __attribute__ ((__vector_size__ (16)));
typedef char __v16qi __attribute__ ((__vector_size__ (16)));
typedef unsigned char __v16qu __attribute__ ((__vector_size__ (16)));
typedef long long __m128i __attribute__ ((__vector_size__ (16), __may_alias__));
typedef double __m128d __attribute__ ((__vector_size__ (16), __may_alias__));
typedef long long __m128i_u __attribute__ ((__vector_size__ (16), __may_alias__, __aligned__ (1)));
typedef double __m128d_u __attribute__ ((__vector_size__ (16), __may_alias__, __aligned__ (1)));
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_sd (double __F)
{
  return __extension__ (__m128d){ __F, 0.0 };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_pd (double __F)
{
  return __extension__ (__m128d){ __F, __F };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pd1 (double __F)
{
  return _mm_set1_pd (__F);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_pd (double __W, double __X)
{
  return __extension__ (__m128d){ __X, __W };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_pd (double __W, double __X)
{
  return __extension__ (__m128d){ __W, __X };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_undefined_pd (void)
{
  __m128d __Y = __Y;
  return __Y;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setzero_pd (void)
{
  return __extension__ (__m128d){ 0.0, 0.0 };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_move_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_movsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_pd (double const *__P)
{
  return *(__m128d *)__P;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadu_pd (double const *__P)
{
  return *(__m128d_u *)__P;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load1_pd (double const *__P)
{
  return _mm_set1_pd (*__P);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_sd (double const *__P)
{
  return _mm_set_sd (*__P);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_pd1 (double const *__P)
{
  return _mm_load1_pd (__P);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadr_pd (double const *__P)
{
  __m128d __tmp = _mm_load_pd (__P);
  return __builtin_ia32_shufpd (__tmp, __tmp, (((0) << 1) | (1)));
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_pd (double *__P, __m128d __A)
{
  *(__m128d *)__P = __A;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storeu_pd (double *__P, __m128d __A)
{
  *(__m128d_u *)__P = __A;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_sd (double *__P, __m128d __A)
{
  *__P = ((__v2df)__A)[0];
}
extern __inline double __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_f64 (__m128d __A)
{
  return ((__v2df)__A)[0];
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storel_pd (double *__P, __m128d __A)
{
  _mm_store_sd (__P, __A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storeh_pd (double *__P, __m128d __A)
{
  *__P = ((__v2df)__A)[1];
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store1_pd (double *__P, __m128d __A)
{
  _mm_store_pd (__P, __builtin_ia32_shufpd (__A, __A, (((0) << 1) | (0))));
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_pd1 (double *__P, __m128d __A)
{
  _mm_store1_pd (__P, __A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storer_pd (double *__P, __m128d __A)
{
  _mm_store_pd (__P, __builtin_ia32_shufpd (__A, __A, (((0) << 1) | (1))));
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi128_si32 (__m128i __A)
{
  return __builtin_ia32_vec_ext_v4si ((__v4si)__A, 0);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi128_si64 (__m128i __A)
{
  return ((__v2di)__A)[0];
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi128_si64x (__m128i __A)
{
  return ((__v2di)__A)[0];
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_pd (__m128d __A, __m128d __B)
{
  return (__m128d) ((__v2df)__A + (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_addsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_pd (__m128d __A, __m128d __B)
{
  return (__m128d) ((__v2df)__A - (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_subsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_pd (__m128d __A, __m128d __B)
{
  return (__m128d) ((__v2df)__A * (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_mulsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_div_pd (__m128d __A, __m128d __B)
{
  return (__m128d) ((__v2df)__A / (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_div_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_divsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sqrt_pd (__m128d __A)
{
  return (__m128d)__builtin_ia32_sqrtpd ((__v2df)__A);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sqrt_sd (__m128d __A, __m128d __B)
{
  __v2df __tmp = __builtin_ia32_movsd ((__v2df)__A, (__v2df)__B);
  return (__m128d)__builtin_ia32_sqrtsd ((__v2df)__tmp);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_minpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_minsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_maxpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_maxsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_and_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_andpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_andnot_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_andnpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_or_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_orpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_xor_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_xorpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpeqpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpltpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmplepd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpgtpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpgepd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpneqpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnlt_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpnltpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnle_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpnlepd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpngt_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpngtpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnge_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpngepd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpord_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpordpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpunord_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpunordpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpeqsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpltsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmplesd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_movsd ((__v2df) __A,
      (__v2df)
      __builtin_ia32_cmpltsd ((__v2df) __B,
         (__v2df)
         __A));
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_movsd ((__v2df) __A,
      (__v2df)
      __builtin_ia32_cmplesd ((__v2df) __B,
         (__v2df)
         __A));
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpneqsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnlt_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpnltsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnle_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpnlesd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpngt_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_movsd ((__v2df) __A,
      (__v2df)
      __builtin_ia32_cmpnltsd ((__v2df) __B,
          (__v2df)
          __A));
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpnge_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_movsd ((__v2df) __A,
      (__v2df)
      __builtin_ia32_cmpnlesd ((__v2df) __B,
          (__v2df)
          __A));
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpord_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpordsd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpunord_sd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_cmpunordsd ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comieq_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdeq ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comilt_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdlt ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comile_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdle ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comigt_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdgt ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comige_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdge ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_comineq_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_comisdneq ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomieq_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdeq ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomilt_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdlt ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomile_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdle ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomigt_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdgt ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomige_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdge ((__v2df)__A, (__v2df)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_ucomineq_sd (__m128d __A, __m128d __B)
{
  return __builtin_ia32_ucomisdneq ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_epi64x (long long __q1, long long __q0)
{
  return __extension__ (__m128i)(__v2di){ __q0, __q1 };
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_epi64 (__m64 __q1, __m64 __q0)
{
  return _mm_set_epi64x ((long long)__q1, (long long)__q0);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_epi32 (int __q3, int __q2, int __q1, int __q0)
{
  return __extension__ (__m128i)(__v4si){ __q0, __q1, __q2, __q3 };
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_epi16 (short __q7, short __q6, short __q5, short __q4,
        short __q3, short __q2, short __q1, short __q0)
{
  return __extension__ (__m128i)(__v8hi){
    __q0, __q1, __q2, __q3, __q4, __q5, __q6, __q7 };
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set_epi8 (char __q15, char __q14, char __q13, char __q12,
       char __q11, char __q10, char __q09, char __q08,
       char __q07, char __q06, char __q05, char __q04,
       char __q03, char __q02, char __q01, char __q00)
{
  return __extension__ (__m128i)(__v16qi){
    __q00, __q01, __q02, __q03, __q04, __q05, __q06, __q07,
    __q08, __q09, __q10, __q11, __q12, __q13, __q14, __q15
  };
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_epi64x (long long __A)
{
  return _mm_set_epi64x (__A, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_epi64 (__m64 __A)
{
  return _mm_set_epi64 (__A, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_epi32 (int __A)
{
  return _mm_set_epi32 (__A, __A, __A, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_epi16 (short __A)
{
  return _mm_set_epi16 (__A, __A, __A, __A, __A, __A, __A, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_set1_epi8 (char __A)
{
  return _mm_set_epi8 (__A, __A, __A, __A, __A, __A, __A, __A,
         __A, __A, __A, __A, __A, __A, __A, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_epi64 (__m64 __q0, __m64 __q1)
{
  return _mm_set_epi64 (__q1, __q0);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_epi32 (int __q0, int __q1, int __q2, int __q3)
{
  return _mm_set_epi32 (__q3, __q2, __q1, __q0);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_epi16 (short __q0, short __q1, short __q2, short __q3,
         short __q4, short __q5, short __q6, short __q7)
{
  return _mm_set_epi16 (__q7, __q6, __q5, __q4, __q3, __q2, __q1, __q0);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setr_epi8 (char __q00, char __q01, char __q02, char __q03,
        char __q04, char __q05, char __q06, char __q07,
        char __q08, char __q09, char __q10, char __q11,
        char __q12, char __q13, char __q14, char __q15)
{
  return _mm_set_epi8 (__q15, __q14, __q13, __q12, __q11, __q10, __q09, __q08,
         __q07, __q06, __q05, __q04, __q03, __q02, __q01, __q00);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_load_si128 (__m128i const *__P)
{
  return *__P;
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadu_si128 (__m128i_u const *__P)
{
  return *__P;
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadl_epi64 (__m128i_u const *__P)
{
  return _mm_set_epi64 ((__m64)0LL, *(__m64_u *)__P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_si128 (__m128i *__P, __m128i __B)
{
  *__P = __B;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storeu_si128 (__m128i_u *__P, __m128i __B)
{
  *__P = __B;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_storel_epi64 (__m128i_u *__P, __m128i __B)
{
  *(__m64_u *)__P = (__m64) ((__v2di)__B)[0];
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movepi64_pi64 (__m128i __B)
{
  return (__m64) ((__v2di)__B)[0];
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movpi64_epi64 (__m64 __A)
{
  return _mm_set_epi64 ((__m64)0LL, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_move_epi64 (__m128i __A)
{
  return (__m128i)__builtin_ia32_movq128 ((__v2di) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_undefined_si128 (void)
{
  __m128i __Y = __Y;
  return __Y;
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_setzero_si128 (void)
{
  return __extension__ (__m128i)(__v4si){ 0, 0, 0, 0 };
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi32_pd (__m128i __A)
{
  return (__m128d)__builtin_ia32_cvtdq2pd ((__v4si) __A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi32_ps (__m128i __A)
{
  return (__m128)__builtin_ia32_cvtdq2ps ((__v4si) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_epi32 (__m128d __A)
{
  return (__m128i)__builtin_ia32_cvtpd2dq ((__v2df) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_pi32 (__m128d __A)
{
  return (__m64)__builtin_ia32_cvtpd2pi ((__v2df) __A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_ps (__m128d __A)
{
  return (__m128)__builtin_ia32_cvtpd2ps ((__v2df) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttpd_epi32 (__m128d __A)
{
  return (__m128i)__builtin_ia32_cvttpd2dq ((__v2df) __A);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttpd_pi32 (__m128d __A)
{
  return (__m64)__builtin_ia32_cvttpd2pi ((__v2df) __A);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpi32_pd (__m64 __A)
{
  return (__m128d)__builtin_ia32_cvtpi2pd ((__v2si) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_epi32 (__m128 __A)
{
  return (__m128i)__builtin_ia32_cvtps2dq ((__v4sf) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttps_epi32 (__m128 __A)
{
  return (__m128i)__builtin_ia32_cvttps2dq ((__v4sf) __A);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_pd (__m128 __A)
{
  return (__m128d)__builtin_ia32_cvtps2pd ((__v4sf) __A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_si32 (__m128d __A)
{
  return __builtin_ia32_cvtsd2si ((__v2df) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_si64 (__m128d __A)
{
  return __builtin_ia32_cvtsd2si64 ((__v2df) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_si64x (__m128d __A)
{
  return __builtin_ia32_cvtsd2si64 ((__v2df) __A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_si32 (__m128d __A)
{
  return __builtin_ia32_cvttsd2si ((__v2df) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_si64 (__m128d __A)
{
  return __builtin_ia32_cvttsd2si64 ((__v2df) __A);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_si64x (__m128d __A)
{
  return __builtin_ia32_cvttsd2si64 ((__v2df) __A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_ss (__m128 __A, __m128d __B)
{
  return (__m128)__builtin_ia32_cvtsd2ss ((__v4sf) __A, (__v2df) __B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi32_sd (__m128d __A, int __B)
{
  return (__m128d)__builtin_ia32_cvtsi2sd ((__v2df) __A, __B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_sd (__m128d __A, long long __B)
{
  return (__m128d)__builtin_ia32_cvtsi642sd ((__v2df) __A, __B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64x_sd (__m128d __A, long long __B)
{
  return (__m128d)__builtin_ia32_cvtsi642sd ((__v2df) __A, __B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_sd (__m128d __A, __m128 __B)
{
  return (__m128d)__builtin_ia32_cvtss2sd ((__v2df) __A, (__v4sf)__B);
}
# 918 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_unpckhpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_pd (__m128d __A, __m128d __B)
{
  return (__m128d)__builtin_ia32_unpcklpd ((__v2df)__A, (__v2df)__B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadh_pd (__m128d __A, double const *__B)
{
  return (__m128d)__builtin_ia32_loadhpd ((__v2df)__A, __B);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loadl_pd (__m128d __A, double const *__B)
{
  return (__m128d)__builtin_ia32_loadlpd ((__v2df)__A, __B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movemask_pd (__m128d __A)
{
  return __builtin_ia32_movmskpd ((__v2df)__A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packs_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_packsswb128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packs_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_packssdw128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packus_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_packuswb128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpckhbw128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpckhwd128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpckhdq128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpackhi_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpckhqdq128 ((__v2di)__A, (__v2di)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpcklbw128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpcklwd128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpckldq128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_unpacklo_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_punpcklqdq128 ((__v2di)__A, (__v2di)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v16qu)__A + (__v16qu)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hu)__A + (__v8hu)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v4su)__A + (__v4su)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_add_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du)__A + (__v2du)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_paddsb128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_paddsw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_paddusb128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_adds_epu16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_paddusw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v16qu)__A - (__v16qu)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hu)__A - (__v8hu)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v4su)__A - (__v4su)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sub_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du)__A - (__v2du)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psubsb128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psubsw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psubusb128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_subs_epu16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psubusw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_madd_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmaddwd128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhi_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmulhw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mullo_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hu)__A * (__v8hu)__B);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_su32 (__m64 __A, __m64 __B)
{
  return (__m64)__builtin_ia32_pmuludq ((__v2si)__A, (__v2si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmuludq128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_epi16 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psllwi128 ((__v8hi)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_epi32 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_pslldi128 ((__v4si)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_slli_epi64 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psllqi128 ((__v2di)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srai_epi16 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psrawi128 ((__v8hi)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srai_epi32 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psradi128 ((__v4si)__A, __B);
}
# 1205 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_epi16 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psrlwi128 ((__v8hi)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_epi32 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psrldi128 ((__v4si)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srli_epi64 (__m128i __A, int __B)
{
  return (__m128i)__builtin_ia32_psrlqi128 ((__v2di)__A, __B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psllw128((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pslld128((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sll_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psllq128((__v2di)__A, (__v2di)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sra_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psraw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sra_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psrad128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psrlw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psrld128 ((__v4si)__A, (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_srl_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psrlq128 ((__v2di)__A, (__v2di)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_and_si128 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du)__A & (__v2du)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_andnot_si128 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pandn128 ((__v2di)__A, (__v2di)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_or_si128 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du)__A | (__v2du)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_xor_si128 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du)__A ^ (__v2du)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v16qi)__A == (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hi)__A == (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v4si)__A == (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v16qi)__A < (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hi)__A < (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v4si)__A < (__v4si)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v16qi)__A > (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v8hi)__A > (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v4si)__A > (__v4si)__B);
}
# 1369 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmaxsw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmaxub128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pminsw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pminub128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movemask_epi8 (__m128i __A)
{
  return __builtin_ia32_pmovmskb128 ((__v16qi)__A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhi_epu16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pmulhuw128 ((__v8hi)__A, (__v8hi)__B);
}
# 1432 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/emmintrin.h" 3 4
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskmoveu_si128 (__m128i __A, __m128i __B, char *__C)
{
  __builtin_ia32_maskmovdqu ((__v16qi)__A, (__v16qi)__B, __C);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_avg_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pavgb128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_avg_epu16 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_pavgw128 ((__v8hi)__A, (__v8hi)__B);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sad_epu8 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_psadbw128 ((__v16qi)__A, (__v16qi)__B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_si32 (int *__A, int __B)
{
  __builtin_ia32_movnti (__A, __B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_si64 (long long int *__A, long long int __B)
{
  __builtin_ia32_movnti64 (__A, __B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_si128 (__m128i *__A, __m128i __B)
{
  __builtin_ia32_movntdq ((__v2di *)__A, (__v2di)__B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_pd (double *__A, __m128d __B)
{
  __builtin_ia32_movntpd (__A, (__v2df)__B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_clflush (void const *__A)
{
  __builtin_ia32_clflush (__A);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_lfence (void)
{
  __builtin_ia32_lfence ();
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mfence (void)
{
  __builtin_ia32_mfence ();
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi32_si128 (int __A)
{
  return _mm_set_epi32 (0, 0, 0, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64_si128 (long long __A)
{
  return _mm_set_epi64x (0, __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsi64x_si128 (long long __A)
{
  return _mm_set_epi64x (0, __A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castpd_ps(__m128d __A)
{
  return (__m128) __A;
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castpd_si128(__m128d __A)
{
  return (__m128i) __A;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castps_pd(__m128 __A)
{
  return (__m128d) __A;
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castps_si128(__m128 __A)
{
  return (__m128i) __A;
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castsi128_ps(__m128i __A)
{
  return (__m128) __A;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_castsi128_pd(__m128i __A)
{
  return (__m128d) __A;
}
# 1253 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 2 3 4
# 1264 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xmmintrin.h" 3 4
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_pause (void)
{
  __builtin_ia32_pause ();
}
# 30 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/pmmintrin.h" 1 3 4
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/pmmintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("sse3")
# 49 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/pmmintrin.h" 3 4
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_addsub_ps (__m128 __X, __m128 __Y)
{
  return (__m128) __builtin_ia32_addsubps ((__v4sf)__X, (__v4sf)__Y);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_ps (__m128 __X, __m128 __Y)
{
  return (__m128) __builtin_ia32_haddps ((__v4sf)__X, (__v4sf)__Y);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_ps (__m128 __X, __m128 __Y)
{
  return (__m128) __builtin_ia32_hsubps ((__v4sf)__X, (__v4sf)__Y);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movehdup_ps (__m128 __X)
{
  return (__m128) __builtin_ia32_movshdup ((__v4sf)__X);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_moveldup_ps (__m128 __X)
{
  return (__m128) __builtin_ia32_movsldup ((__v4sf)__X);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_addsub_pd (__m128d __X, __m128d __Y)
{
  return (__m128d) __builtin_ia32_addsubpd ((__v2df)__X, (__v2df)__Y);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_pd (__m128d __X, __m128d __Y)
{
  return (__m128d) __builtin_ia32_haddpd ((__v2df)__X, (__v2df)__Y);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_pd (__m128d __X, __m128d __Y)
{
  return (__m128d) __builtin_ia32_hsubpd ((__v2df)__X, (__v2df)__Y);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_loaddup_pd (double const *__P)
{
  return _mm_load1_pd (__P);
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_movedup_pd (__m128d __X)
{
  return ((__m128d)__builtin_ia32_shufpd ((__v2df)(__m128d)(__X), (__v2df)(__m128d)(__X), (int)((((0) << 1) | (0)))));
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_lddqu_si128 (__m128i const *__P)
{
  return (__m128i) __builtin_ia32_lddqu ((char const *)__P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_monitor (void const * __P, unsigned int __E, unsigned int __H)
{
  __builtin_ia32_monitor (__P, __E, __H);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mwait (unsigned int __E, unsigned int __H)
{
  __builtin_ia32_mwait (__E, __H);
}
#pragma GCC pop_options
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/tmmintrin.h" 1 3 4
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/tmmintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("ssse3")
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phaddw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phaddd128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadds_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phaddsw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phaddw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadd_pi32 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phaddd ((__v2si)__X, (__v2si)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hadds_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phaddsw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phsubw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phsubd128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsubs_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_phsubsw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phsubw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsub_pi32 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phsubd ((__v2si)__X, (__v2si)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_hsubs_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_phsubsw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maddubs_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaddubsw128 ((__v16qi)__X, (__v16qi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maddubs_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_pmaddubsw ((__v8qi)__X, (__v8qi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhrs_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmulhrsw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mulhrs_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_pmulhrsw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shuffle_epi8 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pshufb128 ((__v16qi)__X, (__v16qi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shuffle_pi8 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_pshufb ((__v8qi)__X, (__v8qi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_epi8 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psignb128 ((__v16qi)__X, (__v16qi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_epi16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psignw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psignd128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_pi8 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_psignb ((__v8qi)__X, (__v8qi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_pi16 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_psignw ((__v4hi)__X, (__v4hi)__Y);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_sign_pi32 (__m64 __X, __m64 __Y)
{
  return (__m64) __builtin_ia32_psignd ((__v2si)__X, (__v2si)__Y);
}
# 208 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/tmmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_epi8 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pabsb128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_epi16 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pabsw128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pabsd128 ((__v4si)__X);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_pi8 (__m64 __X)
{
  return (__m64) __builtin_ia32_pabsb ((__v8qi)__X);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_pi16 (__m64 __X)
{
  return (__m64) __builtin_ia32_pabsw ((__v4hi)__X);
}
extern __inline __m64 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_pi32 (__m64 __X)
{
  return (__m64) __builtin_ia32_pabsd ((__v2si)__X);
}
#pragma GCC pop_options
# 36 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 1 3 4
# 35 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("sse4.1")
# 66 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testz_si128 (__m128i __M, __m128i __V)
{
  return __builtin_ia32_ptestz128 ((__v2di)__M, (__v2di)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testc_si128 (__m128i __M, __m128i __V)
{
  return __builtin_ia32_ptestc128 ((__v2di)__M, (__v2di)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testnzc_si128 (__m128i __M, __m128i __V)
{
  return __builtin_ia32_ptestnzc128 ((__v2di)__M, (__v2di)__V);
}
# 178 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_blendv_epi8 (__m128i __X, __m128i __Y, __m128i __M)
{
  return (__m128i) __builtin_ia32_pblendvb128 ((__v16qi)__X,
            (__v16qi)__Y,
            (__v16qi)__M);
}
# 203 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_blendv_ps (__m128 __X, __m128 __Y, __m128 __M)
{
  return (__m128) __builtin_ia32_blendvps ((__v4sf)__X,
        (__v4sf)__Y,
        (__v4sf)__M);
}
# 228 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_blendv_pd (__m128d __X, __m128d __Y, __m128d __M)
{
  return (__m128d) __builtin_ia32_blendvpd ((__v2df)__X,
         (__v2df)__Y,
         (__v2df)__M);
}
# 267 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi64 (__m128i __X, __m128i __Y)
{
  return (__m128i) ((__v2di)__X == (__v2di)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epi8 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pminsb128 ((__v16qi)__X, (__v16qi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epi8 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaxsb128 ((__v16qi)__X, (__v16qi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epu16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pminuw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epu16 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaxuw128 ((__v8hi)__X, (__v8hi)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pminsd128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaxsd128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epu32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pminud128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epu32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaxud128 ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mullo_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) ((__v4su)__X * (__v4su)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mul_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmuldq128 ((__v4si)__X, (__v4si)__Y);
}
# 473 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_minpos_epu16 (__m128i __X)
{
  return (__m128i) __builtin_ia32_phminposuw128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi8_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxbd128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi16_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxwd128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi8_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxbq128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi32_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxdq128 ((__v4si)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi16_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxwq128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi8_epi16 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxbw128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu8_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxbd128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu16_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxwd128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu8_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxbq128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu32_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxdq128 ((__v4si)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu16_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxwq128 ((__v8hi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu8_epi16 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxbw128 ((__v16qi)__X);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_packus_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_packusdw128 ((__v4si)__X, (__v4si)__Y);
}
# 581 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_stream_load_si128 (__m128i *__X)
{
  return (__m128i) __builtin_ia32_movntdqa ((__v2di *) __X);
}
#pragma GCC push_options
#pragma GCC target("sse4.2")
# 795 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 3 4
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi64 (__m128i __X, __m128i __Y)
{
  return (__m128i) ((__v2di)__X > (__v2di)__Y);
}
#pragma GCC pop_options
#pragma GCC pop_options
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/popcntintrin.h" 1 3 4
# 28 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/popcntintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("popcnt")
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_u32 (unsigned int __X)
{
  return __builtin_popcount (__X);
}
extern __inline long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_u64 (unsigned long long __X)
{
  return __builtin_popcountll (__X);
}
#pragma GCC pop_options
# 812 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/smmintrin.h" 2 3 4
#pragma GCC push_options
#pragma GCC target("sse4.1")
#pragma GCC push_options
#pragma GCC target("sse4.2")
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_crc32_u8 (unsigned int __C, unsigned char __V)
{
  return __builtin_ia32_crc32qi (__C, __V);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_crc32_u16 (unsigned int __C, unsigned short __V)
{
  return __builtin_ia32_crc32hi (__C, __V);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_crc32_u32 (unsigned int __C, unsigned int __V)
{
  return __builtin_ia32_crc32si (__C, __V);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_crc32_u64 (unsigned long long __C, unsigned long long __V)
{
  return __builtin_ia32_crc32di (__C, __V);
}
#pragma GCC pop_options
#pragma GCC pop_options
# 38 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/wmmintrin.h" 1 3 4
# 36 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/wmmintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("aes,sse2")
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesdec_si128 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_aesdec128 ((__v2di)__X, (__v2di)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesdeclast_si128 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_aesdeclast128 ((__v2di)__X,
       (__v2di)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesenc_si128 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_aesenc128 ((__v2di)__X, (__v2di)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesenclast_si128 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_aesenclast128 ((__v2di)__X, (__v2di)__Y);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesimc_si128 (__m128i __X)
{
  return (__m128i) __builtin_ia32_aesimc128 ((__v2di)__X);
}
# 99 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/wmmintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("pclmul,sse2")
# 129 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/wmmintrin.h" 3 4
#pragma GCC pop_options
# 40 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 1 3 4
# 35 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx")
typedef double __v4df __attribute__ ((__vector_size__ (32)));
typedef float __v8sf __attribute__ ((__vector_size__ (32)));
typedef long long __v4di __attribute__ ((__vector_size__ (32)));
typedef unsigned long long __v4du __attribute__ ((__vector_size__ (32)));
typedef int __v8si __attribute__ ((__vector_size__ (32)));
typedef unsigned int __v8su __attribute__ ((__vector_size__ (32)));
typedef short __v16hi __attribute__ ((__vector_size__ (32)));
typedef unsigned short __v16hu __attribute__ ((__vector_size__ (32)));
typedef char __v32qi __attribute__ ((__vector_size__ (32)));
typedef unsigned char __v32qu __attribute__ ((__vector_size__ (32)));
typedef float __m256 __attribute__ ((__vector_size__ (32),
         __may_alias__));
typedef long long __m256i __attribute__ ((__vector_size__ (32),
       __may_alias__));
typedef double __m256d __attribute__ ((__vector_size__ (32),
           __may_alias__));
typedef float __m256_u __attribute__ ((__vector_size__ (32),
           __may_alias__,
           __aligned__ (1)));
typedef long long __m256i_u __attribute__ ((__vector_size__ (32),
         __may_alias__,
         __aligned__ (1)));
typedef double __m256d_u __attribute__ ((__vector_size__ (32),
      __may_alias__,
      __aligned__ (1)));
# 139 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_pd (__m256d __A, __m256d __B)
{
  return (__m256d) ((__v4df)__A + (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_ps (__m256 __A, __m256 __B)
{
  return (__m256) ((__v8sf)__A + (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_addsub_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_addsubpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_addsub_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_addsubps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_and_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_andpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_and_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_andps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_andnot_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_andnpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_andnot_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_andnps256 ((__v8sf)__A, (__v8sf)__B);
}
# 217 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_blendv_pd (__m256d __X, __m256d __Y, __m256d __M)
{
  return (__m256d) __builtin_ia32_blendvpd256 ((__v4df)__X,
            (__v4df)__Y,
            (__v4df)__M);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_blendv_ps (__m256 __X, __m256 __Y, __m256 __M)
{
  return (__m256) __builtin_ia32_blendvps256 ((__v8sf)__X,
           (__v8sf)__Y,
           (__v8sf)__M);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_div_pd (__m256d __A, __m256d __B)
{
  return (__m256d) ((__v4df)__A / (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_div_ps (__m256 __A, __m256 __B)
{
  return (__m256) ((__v8sf)__A / (__v8sf)__B);
}
# 262 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hadd_pd (__m256d __X, __m256d __Y)
{
  return (__m256d) __builtin_ia32_haddpd256 ((__v4df)__X, (__v4df)__Y);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hadd_ps (__m256 __X, __m256 __Y)
{
  return (__m256) __builtin_ia32_haddps256 ((__v8sf)__X, (__v8sf)__Y);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hsub_pd (__m256d __X, __m256d __Y)
{
  return (__m256d) __builtin_ia32_hsubpd256 ((__v4df)__X, (__v4df)__Y);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hsub_ps (__m256 __X, __m256 __Y)
{
  return (__m256) __builtin_ia32_hsubps256 ((__v8sf)__X, (__v8sf)__Y);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_maxpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_maxps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_minpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_minps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mul_pd (__m256d __A, __m256d __B)
{
  return (__m256d) ((__v4df)__A * (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mul_ps (__m256 __A, __m256 __B)
{
  return (__m256) ((__v8sf)__A * (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_or_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_orpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_or_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_orps256 ((__v8sf)__A, (__v8sf)__B);
}
# 358 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_pd (__m256d __A, __m256d __B)
{
  return (__m256d) ((__v4df)__A - (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_ps (__m256 __A, __m256 __B)
{
  return (__m256) ((__v8sf)__A - (__v8sf)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_xor_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_xorpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_xor_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_xorps256 ((__v8sf)__A, (__v8sf)__B);
}
# 446 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi32_pd (__m128i __A)
{
  return (__m256d)__builtin_ia32_cvtdq2pd256 ((__v4si) __A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi32_ps (__m256i __A)
{
  return (__m256)__builtin_ia32_cvtdq2ps256 ((__v8si) __A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtpd_ps (__m256d __A)
{
  return (__m128)__builtin_ia32_cvtpd2ps256 ((__v4df) __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtps_epi32 (__m256 __A)
{
  return (__m256i)__builtin_ia32_cvtps2dq256 ((__v8sf) __A);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtps_pd (__m128 __A)
{
  return (__m256d)__builtin_ia32_cvtps2pd256 ((__v4sf) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttpd_epi32 (__m256d __A)
{
  return (__m128i)__builtin_ia32_cvttpd2dq256 ((__v4df) __A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtpd_epi32 (__m256d __A)
{
  return (__m128i)__builtin_ia32_cvtpd2dq256 ((__v4df) __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttps_epi32 (__m256 __A)
{
  return (__m256i)__builtin_ia32_cvttps2dq256 ((__v8sf) __A);
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsd_f64 (__m256d __A)
{
  return __A[0];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtss_f32 (__m256 __A)
{
  return __A[0];
}
# 600 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_zeroall (void)
{
  __builtin_ia32_vzeroall ();
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_zeroupper (void)
{
  __builtin_ia32_vzeroupper ();
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutevar_pd (__m128d __A, __m128i __C)
{
  return (__m128d) __builtin_ia32_vpermilvarpd ((__v2df)__A,
      (__v2di)__C);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutevar_pd (__m256d __A, __m256i __C)
{
  return (__m256d) __builtin_ia32_vpermilvarpd256 ((__v4df)__A,
         (__v4di)__C);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutevar_ps (__m128 __A, __m128i __C)
{
  return (__m128) __builtin_ia32_vpermilvarps ((__v4sf)__A,
            (__v4si)__C);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutevar_ps (__m256 __A, __m256i __C)
{
  return (__m256) __builtin_ia32_vpermilvarps256 ((__v8sf)__A,
        (__v8si)__C);
}
# 719 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcast_ss (float const *__X)
{
  return (__m128) __builtin_ia32_vbroadcastss (__X);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_sd (double const *__X)
{
  return (__m256d) __builtin_ia32_vbroadcastsd256 (__X);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_ss (float const *__X)
{
  return (__m256) __builtin_ia32_vbroadcastss256 (__X);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_pd (__m128d const *__X)
{
  return (__m256d) __builtin_ia32_vbroadcastf128_pd256 (__X);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_ps (__m128 const *__X)
{
  return (__m256) __builtin_ia32_vbroadcastf128_ps256 (__X);
}
# 858 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_load_pd (double const *__P)
{
  return *(__m256d *)__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_store_pd (double *__P, __m256d __A)
{
  *(__m256d *)__P = __A;
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_load_ps (float const *__P)
{
  return *(__m256 *)__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_store_ps (float *__P, __m256 __A)
{
  *(__m256 *)__P = __A;
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_loadu_pd (double const *__P)
{
  return *(__m256d_u *)__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_storeu_pd (double *__P, __m256d __A)
{
  *(__m256d_u *)__P = __A;
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_loadu_ps (float const *__P)
{
  return *(__m256_u *)__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_storeu_ps (float *__P, __m256 __A)
{
  *(__m256_u *)__P = __A;
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_load_si256 (__m256i const *__P)
{
  return *__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_store_si256 (__m256i *__P, __m256i __A)
{
  *__P = __A;
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_loadu_si256 (__m256i_u const *__P)
{
  return *__P;
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_storeu_si256 (__m256i_u *__P, __m256i __A)
{
  *__P = __A;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskload_pd (double const *__P, __m128i __M)
{
  return (__m128d) __builtin_ia32_maskloadpd ((const __v2df *)__P,
           (__v2di)__M);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskstore_pd (double *__P, __m128i __M, __m128d __A)
{
  __builtin_ia32_maskstorepd ((__v2df *)__P, (__v2di)__M, (__v2df)__A);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskload_pd (double const *__P, __m256i __M)
{
  return (__m256d) __builtin_ia32_maskloadpd256 ((const __v4df *)__P,
       (__v4di)__M);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskstore_pd (double *__P, __m256i __M, __m256d __A)
{
  __builtin_ia32_maskstorepd256 ((__v4df *)__P, (__v4di)__M, (__v4df)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskload_ps (float const *__P, __m128i __M)
{
  return (__m128) __builtin_ia32_maskloadps ((const __v4sf *)__P,
          (__v4si)__M);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskstore_ps (float *__P, __m128i __M, __m128 __A)
{
  __builtin_ia32_maskstoreps ((__v4sf *)__P, (__v4si)__M, (__v4sf)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskload_ps (float const *__P, __m256i __M)
{
  return (__m256) __builtin_ia32_maskloadps256 ((const __v8sf *)__P,
      (__v8si)__M);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskstore_ps (float *__P, __m256i __M, __m256 __A)
{
  __builtin_ia32_maskstoreps256 ((__v8sf *)__P, (__v8si)__M, (__v8sf)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movehdup_ps (__m256 __X)
{
  return (__m256) __builtin_ia32_movshdup256 ((__v8sf)__X);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_moveldup_ps (__m256 __X)
{
  return (__m256) __builtin_ia32_movsldup256 ((__v8sf)__X);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movedup_pd (__m256d __X)
{
  return (__m256d) __builtin_ia32_movddup256 ((__v4df)__X);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_lddqu_si256 (__m256i const *__P)
{
  return (__m256i) __builtin_ia32_lddqu256 ((char const *)__P);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_stream_si256 (__m256i *__A, __m256i __B)
{
  __builtin_ia32_movntdq256 ((__v4di *)__A, (__v4di)__B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_stream_pd (double *__A, __m256d __B)
{
  __builtin_ia32_movntpd256 (__A, (__v4df)__B);
}
extern __inline void __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_stream_ps (float *__P, __m256 __A)
{
  __builtin_ia32_movntps256 (__P, (__v8sf)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rcp_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_rcpps256 ((__v8sf)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rsqrt_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_rsqrtps256 ((__v8sf)__A);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sqrt_pd (__m256d __A)
{
  return (__m256d) __builtin_ia32_sqrtpd256 ((__v4df)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sqrt_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_sqrtps256 ((__v8sf)__A);
}
# 1073 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avxintrin.h" 3 4
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_unpckhpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_unpcklpd256 ((__v4df)__A, (__v4df)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_unpckhps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_unpcklps256 ((__v8sf)__A, (__v8sf)__B);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testz_pd (__m128d __M, __m128d __V)
{
  return __builtin_ia32_vtestzpd ((__v2df)__M, (__v2df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testc_pd (__m128d __M, __m128d __V)
{
  return __builtin_ia32_vtestcpd ((__v2df)__M, (__v2df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testnzc_pd (__m128d __M, __m128d __V)
{
  return __builtin_ia32_vtestnzcpd ((__v2df)__M, (__v2df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testz_ps (__m128 __M, __m128 __V)
{
  return __builtin_ia32_vtestzps ((__v4sf)__M, (__v4sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testc_ps (__m128 __M, __m128 __V)
{
  return __builtin_ia32_vtestcps ((__v4sf)__M, (__v4sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_testnzc_ps (__m128 __M, __m128 __V)
{
  return __builtin_ia32_vtestnzcps ((__v4sf)__M, (__v4sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testz_pd (__m256d __M, __m256d __V)
{
  return __builtin_ia32_vtestzpd256 ((__v4df)__M, (__v4df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testc_pd (__m256d __M, __m256d __V)
{
  return __builtin_ia32_vtestcpd256 ((__v4df)__M, (__v4df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testnzc_pd (__m256d __M, __m256d __V)
{
  return __builtin_ia32_vtestnzcpd256 ((__v4df)__M, (__v4df)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testz_ps (__m256 __M, __m256 __V)
{
  return __builtin_ia32_vtestzps256 ((__v8sf)__M, (__v8sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testc_ps (__m256 __M, __m256 __V)
{
  return __builtin_ia32_vtestcps256 ((__v8sf)__M, (__v8sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testnzc_ps (__m256 __M, __m256 __V)
{
  return __builtin_ia32_vtestnzcps256 ((__v8sf)__M, (__v8sf)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testz_si256 (__m256i __M, __m256i __V)
{
  return __builtin_ia32_ptestz256 ((__v4di)__M, (__v4di)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testc_si256 (__m256i __M, __m256i __V)
{
  return __builtin_ia32_ptestc256 ((__v4di)__M, (__v4di)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testnzc_si256 (__m256i __M, __m256i __V)
{
  return __builtin_ia32_ptestnzc256 ((__v4di)__M, (__v4di)__V);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movemask_pd (__m256d __A)
{
  return __builtin_ia32_movmskpd256 ((__v4df)__A);
}
extern __inline int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movemask_ps (__m256 __A)
{
  return __builtin_ia32_movmskps256 ((__v8sf)__A);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_undefined_pd (void)
{
  __m256d __Y = __Y;
  return __Y;
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_undefined_ps (void)
{
  __m256 __Y = __Y;
  return __Y;
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_undefined_si256 (void)
{
  __m256i __Y = __Y;
  return __Y;
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setzero_pd (void)
{
  return __extension__ (__m256d){ 0.0, 0.0, 0.0, 0.0 };
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setzero_ps (void)
{
  return __extension__ (__m256){ 0.0, 0.0, 0.0, 0.0,
     0.0, 0.0, 0.0, 0.0 };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setzero_si256 (void)
{
  return __extension__ (__m256i)(__v4di){ 0, 0, 0, 0 };
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_pd (double __A, double __B, double __C, double __D)
{
  return __extension__ (__m256d){ __D, __C, __B, __A };
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_ps (float __A, float __B, float __C, float __D,
        float __E, float __F, float __G, float __H)
{
  return __extension__ (__m256){ __H, __G, __F, __E,
     __D, __C, __B, __A };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_epi32 (int __A, int __B, int __C, int __D,
    int __E, int __F, int __G, int __H)
{
  return __extension__ (__m256i)(__v8si){ __H, __G, __F, __E,
       __D, __C, __B, __A };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_epi16 (short __q15, short __q14, short __q13, short __q12,
    short __q11, short __q10, short __q09, short __q08,
    short __q07, short __q06, short __q05, short __q04,
    short __q03, short __q02, short __q01, short __q00)
{
  return __extension__ (__m256i)(__v16hi){
    __q00, __q01, __q02, __q03, __q04, __q05, __q06, __q07,
    __q08, __q09, __q10, __q11, __q12, __q13, __q14, __q15
  };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_epi8 (char __q31, char __q30, char __q29, char __q28,
    char __q27, char __q26, char __q25, char __q24,
    char __q23, char __q22, char __q21, char __q20,
    char __q19, char __q18, char __q17, char __q16,
    char __q15, char __q14, char __q13, char __q12,
    char __q11, char __q10, char __q09, char __q08,
    char __q07, char __q06, char __q05, char __q04,
    char __q03, char __q02, char __q01, char __q00)
{
  return __extension__ (__m256i)(__v32qi){
    __q00, __q01, __q02, __q03, __q04, __q05, __q06, __q07,
    __q08, __q09, __q10, __q11, __q12, __q13, __q14, __q15,
    __q16, __q17, __q18, __q19, __q20, __q21, __q22, __q23,
    __q24, __q25, __q26, __q27, __q28, __q29, __q30, __q31
  };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_epi64x (long long __A, long long __B, long long __C,
     long long __D)
{
  return __extension__ (__m256i)(__v4di){ __D, __C, __B, __A };
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_pd (double __A)
{
  return __extension__ (__m256d){ __A, __A, __A, __A };
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_ps (float __A)
{
  return __extension__ (__m256){ __A, __A, __A, __A,
     __A, __A, __A, __A };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_epi32 (int __A)
{
  return __extension__ (__m256i)(__v8si){ __A, __A, __A, __A,
       __A, __A, __A, __A };
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_epi16 (short __A)
{
  return _mm256_set_epi16 (__A, __A, __A, __A, __A, __A, __A, __A,
      __A, __A, __A, __A, __A, __A, __A, __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_epi8 (char __A)
{
  return _mm256_set_epi8 (__A, __A, __A, __A, __A, __A, __A, __A,
     __A, __A, __A, __A, __A, __A, __A, __A,
     __A, __A, __A, __A, __A, __A, __A, __A,
     __A, __A, __A, __A, __A, __A, __A, __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set1_epi64x (long long __A)
{
  return __extension__ (__m256i)(__v4di){ __A, __A, __A, __A };
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_pd (double __A, double __B, double __C, double __D)
{
  return _mm256_set_pd (__D, __C, __B, __A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_ps (float __A, float __B, float __C, float __D,
  float __E, float __F, float __G, float __H)
{
  return _mm256_set_ps (__H, __G, __F, __E, __D, __C, __B, __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_epi32 (int __A, int __B, int __C, int __D,
     int __E, int __F, int __G, int __H)
{
  return _mm256_set_epi32 (__H, __G, __F, __E, __D, __C, __B, __A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_epi16 (short __q15, short __q14, short __q13, short __q12,
     short __q11, short __q10, short __q09, short __q08,
     short __q07, short __q06, short __q05, short __q04,
     short __q03, short __q02, short __q01, short __q00)
{
  return _mm256_set_epi16 (__q00, __q01, __q02, __q03,
      __q04, __q05, __q06, __q07,
      __q08, __q09, __q10, __q11,
      __q12, __q13, __q14, __q15);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_epi8 (char __q31, char __q30, char __q29, char __q28,
     char __q27, char __q26, char __q25, char __q24,
     char __q23, char __q22, char __q21, char __q20,
     char __q19, char __q18, char __q17, char __q16,
     char __q15, char __q14, char __q13, char __q12,
     char __q11, char __q10, char __q09, char __q08,
     char __q07, char __q06, char __q05, char __q04,
     char __q03, char __q02, char __q01, char __q00)
{
  return _mm256_set_epi8 (__q00, __q01, __q02, __q03,
     __q04, __q05, __q06, __q07,
     __q08, __q09, __q10, __q11,
     __q12, __q13, __q14, __q15,
     __q16, __q17, __q18, __q19,
     __q20, __q21, __q22, __q23,
     __q24, __q25, __q26, __q27,
     __q28, __q29, __q30, __q31);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_epi64x (long long __A, long long __B, long long __C,
      long long __D)
{
  return _mm256_set_epi64x (__D, __C, __B, __A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castpd_ps (__m256d __A)
{
  return (__m256) __A;
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castpd_si256 (__m256d __A)
{
  return (__m256i) __A;
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castps_pd (__m256 __A)
{
  return (__m256d) __A;
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castps_si256(__m256 __A)
{
  return (__m256i) __A;
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castsi256_ps (__m256i __A)
{
  return (__m256) __A;
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castsi256_pd (__m256i __A)
{
  return (__m256d) __A;
}
extern __inline __m128d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castpd256_pd128 (__m256d __A)
{
  return (__m128d) __builtin_ia32_pd_pd256 ((__v4df)__A);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castps256_ps128 (__m256 __A)
{
  return (__m128) __builtin_ia32_ps_ps256 ((__v8sf)__A);
}
extern __inline __m128i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castsi256_si128 (__m256i __A)
{
  return (__m128i) __builtin_ia32_si_si256 ((__v8si)__A);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castpd128_pd256 (__m128d __A)
{
  return (__m256d) __builtin_ia32_pd256_pd ((__v2df)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castps128_ps256 (__m128 __A)
{
  return (__m256) __builtin_ia32_ps256_ps ((__v4sf)__A);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_castsi128_si256 (__m128i __A)
{
  return (__m256i) __builtin_ia32_si256_si ((__v4si)__A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_m128 ( __m128 __H, __m128 __L)
{
  return ((__m256) __builtin_ia32_vinsertf128_ps256 ((__v8sf)(__m256)(_mm256_castps128_ps256 (__L)), (__v4sf)(__m128)(__H), (int)(1)));
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_m128d (__m128d __H, __m128d __L)
{
  return ((__m256d) __builtin_ia32_vinsertf128_pd256 ((__v4df)(__m256d)(_mm256_castpd128_pd256 (__L)), (__v2df)(__m128d)(__H), (int)(1)));
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_set_m128i (__m128i __H, __m128i __L)
{
  return ((__m256i) __builtin_ia32_vinsertf128_si256 ((__v8si)(__m256i)(_mm256_castsi128_si256 (__L)), (__v4si)(__m128i)(__H), (int)(1)));
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_m128 (__m128 __L, __m128 __H)
{
  return _mm256_set_m128 (__H, __L);
}
extern __inline __m256d __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_m128d (__m128d __L, __m128d __H)
{
  return _mm256_set_m128d (__H, __L);
}
extern __inline __m256i __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_setr_m128i (__m128i __L, __m128i __H)
{
  return _mm256_set_m128i (__H, __L);
}
#pragma GCC pop_options
# 42 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx2")
# 54 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_abs_epi8 (__m256i __A)
{
  return (__m256i)__builtin_ia32_pabsb256 ((__v32qi)__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_abs_epi16 (__m256i __A)
{
  return (__m256i)__builtin_ia32_pabsw256 ((__v16hi)__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_abs_epi32 (__m256i __A)
{
  return (__m256i)__builtin_ia32_pabsd256 ((__v8si)__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_packs_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_packssdw256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_packs_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_packsswb256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_packus_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_packusdw256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_packus_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_packuswb256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v32qu)__A + (__v32qu)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v16hu)__A + (__v16hu)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v8su)__A + (__v8su)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_add_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du)__A + (__v4du)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_adds_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_paddsb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_adds_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_paddsw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_adds_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_paddusb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_adds_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_paddusw256 ((__v16hi)__A, (__v16hi)__B);
}
# 177 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_and_si256 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du)__A & (__v4du)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_andnot_si256 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_andnotsi256 ((__v4di)__A, (__v4di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_avg_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pavgb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_avg_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pavgw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_blendv_epi8 (__m256i __X, __m256i __Y, __m256i __M)
{
  return (__m256i) __builtin_ia32_pblendvb256 ((__v32qi)__X,
            (__v32qi)__Y,
            (__v32qi)__M);
}
# 229 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v32qi)__A == (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v16hi)__A == (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v8si)__A == (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4di)__A == (__v4di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v32qi)__A > (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v16hi)__A > (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v8si)__A > (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4di)__A > (__v4di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hadd_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phaddw256 ((__v16hi)__X,
          (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hadd_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phaddd256 ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hadds_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phaddsw256 ((__v16hi)__X,
           (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hsub_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phsubw256 ((__v16hi)__X,
          (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hsub_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phsubd256 ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_hsubs_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_phsubsw256 ((__v16hi)__X,
           (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maddubs_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmaddubsw256 ((__v32qi)__X,
      (__v32qi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_madd_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaddwd256 ((__v16hi)__A,
          (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxsb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxsw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxsd256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxub256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxuw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epu32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmaxud256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminsb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminsw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminsd256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminub256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminuw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epu32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pminud256 ((__v8si)__A, (__v8si)__B);
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movemask_epi8 (__m256i __A)
{
  return __builtin_ia32_pmovmskb256 ((__v32qi)__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi8_epi16 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxbw256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi8_epi32 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxbd256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi8_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxbq256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi16_epi32 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxwd256 ((__v8hi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi16_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxwq256 ((__v8hi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi32_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxdq256 ((__v4si)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu8_epi16 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxbw256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu8_epi32 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxbd256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu8_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxbq256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu16_epi32 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxwd256 ((__v8hi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu16_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxwq256 ((__v8hi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu32_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxdq256 ((__v4si)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mul_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmuldq256 ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mulhrs_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmulhrsw256 ((__v16hi)__X,
            (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mulhi_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmulhuw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mulhi_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmulhw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mullo_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v16hu)__A * (__v16hu)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mullo_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v8su)__A * (__v8su)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mul_epu32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_pmuludq256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_or_si256 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du)__A | (__v4du)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sad_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_psadbw256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shuffle_epi8 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pshufb256 ((__v32qi)__X,
          (__v32qi)__Y);
}
# 624 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sign_epi8 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psignb256 ((__v32qi)__X, (__v32qi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sign_epi16 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psignw256 ((__v16hi)__X, (__v16hi)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sign_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psignd256 ((__v8si)__X, (__v8si)__Y);
}
# 666 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_slli_epi16 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psllwi256 ((__v16hi)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sll_epi16 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psllw256((__v16hi)__A, (__v8hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_slli_epi32 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_pslldi256 ((__v8si)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sll_epi32 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_pslld256((__v8si)__A, (__v4si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_slli_epi64 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psllqi256 ((__v4di)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sll_epi64 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psllq256((__v4di)__A, (__v2di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srai_epi16 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psrawi256 ((__v16hi)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sra_epi16 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psraw256 ((__v16hi)__A, (__v8hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srai_epi32 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psradi256 ((__v8si)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sra_epi32 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psrad256 ((__v8si)__A, (__v4si)__B);
}
# 757 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srli_epi16 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psrlwi256 ((__v16hi)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srl_epi16 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psrlw256((__v16hi)__A, (__v8hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srli_epi32 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psrldi256 ((__v8si)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srl_epi32 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psrld256((__v8si)__A, (__v4si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srli_epi64 (__m256i __A, int __B)
{
  return (__m256i)__builtin_ia32_psrlqi256 ((__v4di)__A, __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srl_epi64 (__m256i __A, __m128i __B)
{
  return (__m256i)__builtin_ia32_psrlq256((__v4di)__A, (__v2di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v32qu)__A - (__v32qu)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v16hu)__A - (__v16hu)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v8su)__A - (__v8su)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sub_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du)__A - (__v4du)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_subs_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_psubsb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_subs_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_psubsw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_subs_epu8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_psubusb256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_subs_epu16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_psubusw256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpckhbw256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpckhwd256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpckhdq256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpackhi_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpckhqdq256 ((__v4di)__A, (__v4di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpcklbw256 ((__v32qi)__A, (__v32qi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpcklwd256 ((__v16hi)__A, (__v16hi)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpckldq256 ((__v8si)__A, (__v8si)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_unpacklo_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_punpcklqdq256 ((__v4di)__A, (__v4di)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_xor_si256 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du)__A ^ (__v4du)__B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_stream_load_si256 (__m256i const *__X)
{
  return (__m256i) __builtin_ia32_movntdqa256 ((__v4di *) __X);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastss_ps (__m128 __X)
{
  return (__m128) __builtin_ia32_vbroadcastss_ps ((__v4sf)__X);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastss_ps (__m128 __X)
{
  return (__m256) __builtin_ia32_vbroadcastss_ps256 ((__v4sf)__X);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastsd_pd (__m128d __X)
{
  return (__m256d) __builtin_ia32_vbroadcastsd_pd256 ((__v2df)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastsi128_si256 (__m128i __X)
{
  return (__m256i) __builtin_ia32_vbroadcastsi256 ((__v2di)__X);
}
# 983 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastb_epi8 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pbroadcastb256 ((__v16qi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastw_epi16 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pbroadcastw256 ((__v8hi)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastd_epi32 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pbroadcastd256 ((__v4si)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastq_epi64 (__m128i __X)
{
  return (__m256i) __builtin_ia32_pbroadcastq256 ((__v2di)__X);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastb_epi8 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pbroadcastb128 ((__v16qi)__X);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastw_epi16 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pbroadcastw128 ((__v8hi)__X);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastd_epi32 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pbroadcastd128 ((__v4si)__X);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastq_epi64 (__m128i __X)
{
  return (__m128i) __builtin_ia32_pbroadcastq128 ((__v2di)__X);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutevar8x32_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvarsi256 ((__v8si)__X, (__v8si)__Y);
}
# 1058 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutevar8x32_ps (__m256 __X, __m256i __Y)
{
  return (__m256) __builtin_ia32_permvarsf256 ((__v8sf)__X, (__v8si)__Y);
}
# 1116 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskload_epi32 (int const *__X, __m256i __M )
{
  return (__m256i) __builtin_ia32_maskloadd256 ((const __v8si *)__X,
      (__v8si)__M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskload_epi64 (long long const *__X, __m256i __M )
{
  return (__m256i) __builtin_ia32_maskloadq256 ((const __v4di *)__X,
      (__v4di)__M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskload_epi32 (int const *__X, __m128i __M )
{
  return (__m128i) __builtin_ia32_maskloadd ((const __v4si *)__X,
          (__v4si)__M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskload_epi64 (long long const *__X, __m128i __M )
{
  return (__m128i) __builtin_ia32_maskloadq ((const __v2di *)__X,
          (__v2di)__M);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskstore_epi32 (int *__X, __m256i __M, __m256i __Y )
{
  __builtin_ia32_maskstored256 ((__v8si *)__X, (__v8si)__M, (__v8si)__Y);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskstore_epi64 (long long *__X, __m256i __M, __m256i __Y )
{
  __builtin_ia32_maskstoreq256 ((__v4di *)__X, (__v4di)__M, (__v4di)__Y);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskstore_epi32 (int *__X, __m128i __M, __m128i __Y )
{
  __builtin_ia32_maskstored ((__v4si *)__X, (__v4si)__M, (__v4si)__Y);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskstore_epi64 (long long *__X, __m128i __M, __m128i __Y )
{
  __builtin_ia32_maskstoreq (( __v2di *)__X, (__v2di)__M, (__v2di)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sllv_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv8si ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sllv_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv4si ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sllv_epi64 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv4di ((__v4di)__X, (__v4di)__Y);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sllv_epi64 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv2di ((__v2di)__X, (__v2di)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srav_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrav8si ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srav_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrav4si ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srlv_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv8si ((__v8si)__X, (__v8si)__Y);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srlv_epi32 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv4si ((__v4si)__X, (__v4si)__Y);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srlv_epi64 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv4di ((__v4di)__X, (__v4di)__Y);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srlv_epi64 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv2di ((__v2di)__X, (__v2di)__Y);
}
# 1905 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx2intrin.h" 3 4
#pragma GCC pop_options
# 44 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512f")
typedef double __v8df __attribute__ ((__vector_size__ (64)));
typedef float __v16sf __attribute__ ((__vector_size__ (64)));
typedef long long __v8di __attribute__ ((__vector_size__ (64)));
typedef unsigned long long __v8du __attribute__ ((__vector_size__ (64)));
typedef int __v16si __attribute__ ((__vector_size__ (64)));
typedef unsigned int __v16su __attribute__ ((__vector_size__ (64)));
typedef short __v32hi __attribute__ ((__vector_size__ (64)));
typedef unsigned short __v32hu __attribute__ ((__vector_size__ (64)));
typedef char __v64qi __attribute__ ((__vector_size__ (64)));
typedef unsigned char __v64qu __attribute__ ((__vector_size__ (64)));
typedef float __m512 __attribute__ ((__vector_size__ (64), __may_alias__));
typedef long long __m512i __attribute__ ((__vector_size__ (64), __may_alias__));
typedef double __m512d __attribute__ ((__vector_size__ (64), __may_alias__));
typedef float __m512_u __attribute__ ((__vector_size__ (64), __may_alias__, __aligned__ (1)));
typedef long long __m512i_u __attribute__ ((__vector_size__ (64), __may_alias__, __aligned__ (1)));
typedef double __m512d_u __attribute__ ((__vector_size__ (64), __may_alias__, __aligned__ (1)));
typedef unsigned char __mmask8;
typedef unsigned short __mmask16;
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_int2mask (int __M)
{
  return (__mmask16) __M;
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2int (__mmask16 __M)
{
  return (int) __M;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set_epi64 (long long __A, long long __B, long long __C,
    long long __D, long long __E, long long __F,
    long long __G, long long __H)
{
  return __extension__ (__m512i) (__v8di)
  { __H, __G, __F, __E, __D, __C, __B, __A };
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set_epi32 (int __A, int __B, int __C, int __D,
    int __E, int __F, int __G, int __H,
    int __I, int __J, int __K, int __L,
    int __M, int __N, int __O, int __P)
{
  return __extension__ (__m512i)(__v16si)
  { __P, __O, __N, __M, __L, __K, __J, __I,
    __H, __G, __F, __E, __D, __C, __B, __A };
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set_pd (double __A, double __B, double __C, double __D,
        double __E, double __F, double __G, double __H)
{
  return __extension__ (__m512d)
  { __H, __G, __F, __E, __D, __C, __B, __A };
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set_ps (float __A, float __B, float __C, float __D,
        float __E, float __F, float __G, float __H,
        float __I, float __J, float __K, float __L,
        float __M, float __N, float __O, float __P)
{
  return __extension__ (__m512)
  { __P, __O, __N, __M, __L, __K, __J, __I,
    __H, __G, __F, __E, __D, __C, __B, __A };
}
# 134 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_undefined_ps (void)
{
  __m512 __Y = __Y;
  return __Y;
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_undefined_pd (void)
{
  __m512d __Y = __Y;
  return __Y;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_undefined_epi32 (void)
{
  __m512i __Y = __Y;
  return __Y;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_epi8 (char __A)
{
  return __extension__ (__m512i)(__v64qi)
  { __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A };
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_epi16 (short __A)
{
  return __extension__ (__m512i)(__v32hi)
  { __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A,
    __A, __A, __A, __A, __A, __A, __A, __A };
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_pd (double __A)
{
  return (__m512d) __builtin_ia32_broadcastsd512 (__extension__
        (__v2df) { __A, },
        (__v8df)
        _mm512_undefined_pd (),
        (__mmask8) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_ps (float __A)
{
  return (__m512) __builtin_ia32_broadcastss512 (__extension__
       (__v4sf) { __A, },
       (__v16sf)
       _mm512_undefined_ps (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set4_epi32 (int __A, int __B, int __C, int __D)
{
  return __extension__ (__m512i)(__v16si)
  { __D, __C, __B, __A, __D, __C, __B, __A,
    __D, __C, __B, __A, __D, __C, __B, __A };
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set4_epi64 (long long __A, long long __B, long long __C,
     long long __D)
{
  return __extension__ (__m512i) (__v8di)
  { __D, __C, __B, __A, __D, __C, __B, __A };
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set4_pd (double __A, double __B, double __C, double __D)
{
  return __extension__ (__m512d)
  { __D, __C, __B, __A, __D, __C, __B, __A };
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set4_ps (float __A, float __B, float __C, float __D)
{
  return __extension__ (__m512)
  { __D, __C, __B, __A, __D, __C, __B, __A,
    __D, __C, __B, __A, __D, __C, __B, __A };
}
# 258 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_setzero_ps (void)
{
  return __extension__ (__m512){ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 };
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_setzero_pd (void)
{
  return __extension__ (__m512d) { 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 };
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_setzero_epi32 (void)
{
  return __extension__ (__m512i)(__v8di){ 0, 0, 0, 0, 0, 0, 0, 0 };
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_setzero_si512 (void)
{
  return __extension__ (__m512i)(__v8di){ 0, 0, 0, 0, 0, 0, 0, 0 };
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_movapd512_mask ((__v8df) __A,
        (__v8df) __W,
        (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_movapd512_mask ((__v8df) __A,
        (__v8df)
        _mm512_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movaps512_mask ((__v16sf) __A,
       (__v16sf) __W,
       (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movaps512_mask ((__v16sf) __A,
       (__v16sf)
       _mm512_setzero_ps (),
       (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_load_pd (void const *__P)
{
  return *(__m512d *) __P;
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_load_pd (__m512d __W, __mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_loadapd512_mask ((const __v8df *) __P,
         (__v8df) __W,
         (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_load_pd (__mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_loadapd512_mask ((const __v8df *) __P,
         (__v8df)
         _mm512_setzero_pd (),
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_store_pd (void *__P, __m512d __A)
{
  *(__m512d *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_store_pd (void *__P, __mmask8 __U, __m512d __A)
{
  __builtin_ia32_storeapd512_mask ((__v8df *) __P, (__v8df) __A,
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_load_ps (void const *__P)
{
  return *(__m512 *) __P;
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_load_ps (__m512 __W, __mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_loadaps512_mask ((const __v16sf *) __P,
        (__v16sf) __W,
        (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_load_ps (__mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_loadaps512_mask ((const __v16sf *) __P,
        (__v16sf)
        _mm512_setzero_ps (),
        (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_store_ps (void *__P, __m512 __A)
{
  *(__m512 *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_store_ps (void *__P, __mmask16 __U, __m512 __A)
{
  __builtin_ia32_storeaps512_mask ((__v16sf *) __P, (__v16sf) __A,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdqa64_512_mask ((__v8di) __A,
           (__v8di) __W,
           (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdqa64_512_mask ((__v8di) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_load_epi64 (void const *__P)
{
  return *(__m512i *) __P;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_load_epi64 (__m512i __W, __mmask8 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_movdqa64load512_mask ((const __v8di *) __P,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_load_epi64 (__mmask8 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_movdqa64load512_mask ((const __v8di *) __P,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_store_epi64 (void *__P, __m512i __A)
{
  *(__m512i *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_store_epi64 (void *__P, __mmask8 __U, __m512i __A)
{
  __builtin_ia32_movdqa64store512_mask ((__v8di *) __P, (__v8di) __A,
     (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdqa32_512_mask ((__v16si) __A,
           (__v16si) __W,
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdqa32_512_mask ((__v16si) __A,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_load_si512 (void const *__P)
{
  return *(__m512i *) __P;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_load_epi32 (void const *__P)
{
  return *(__m512i *) __P;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_load_epi32 (__m512i __W, __mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_movdqa32load512_mask ((const __v16si *) __P,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_load_epi32 (__mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_movdqa32load512_mask ((const __v16si *) __P,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_store_si512 (void *__P, __m512i __A)
{
  *(__m512i *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_store_epi32 (void *__P, __m512i __A)
{
  *(__m512i *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_store_epi32 (void *__P, __mmask16 __U, __m512i __A)
{
  __builtin_ia32_movdqa32store512_mask ((__v16si *) __P, (__v16si) __A,
     (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mullo_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A * (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mullo_epi32 (__mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulld512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mullo_epi32 (__m512i __W, __mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulld512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sllv_epi32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sllv_epi32 (__m512i __W, __mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sllv_epi32 (__mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srav_epi32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srav_epi32 (__m512i __W, __mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srav_epi32 (__mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srlv_epi32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srlv_epi32 (__m512i __W, __mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srlv_epi32 (__mmask16 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv16si_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A + (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A - (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sllv_epi64 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_undefined_pd (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sllv_epi64 (__m512i __W, __mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sllv_epi64 (__mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psllv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srav_epi64 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srav_epi64 (__m512i __W, __mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srav_epi64 (__mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrav8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srlv_epi64 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srlv_epi64 (__m512i __W, __mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srlv_epi64 (__mmask8 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_psrlv8di_mask ((__v8di) __X,
       (__v8di) __Y,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A + (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mul_epi32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuldq512_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mul_epi32 (__m512i __W, __mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuldq512_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mul_epi32 (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuldq512_mask ((__v16si) __X,
        (__v16si) __Y,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A - (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mul_epu32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuludq512_mask ((__v16si) __X,
         (__v16si) __Y,
         (__v8di)
         _mm512_undefined_epi32 (),
         (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mul_epu32 (__m512i __W, __mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuludq512_mask ((__v16si) __X,
         (__v16si) __Y,
         (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mul_epu32 (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmuludq512_mask ((__v16si) __X,
         (__v16si) __Y,
         (__v8di)
         _mm512_setzero_si512 (),
         __M);
}
# 981 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sll_epi64 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psllq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sll_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psllq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sll_epi64 (__mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psllq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
# 1060 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srl_epi64 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srl_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srl_epi64 (__mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
# 1139 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sra_epi64 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psraq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sra_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psraq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sra_epi64 (__mmask8 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psraq512_mask ((__v8di) __A,
       (__v2di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
# 1218 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sll_epi32 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_pslld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sll_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_pslld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sll_epi32 (__mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_pslld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
# 1297 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srl_epi32 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srl_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srl_epi32 (__mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrld512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
# 1376 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sra_epi32 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrad512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sra_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrad512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sra_epi32 (__mmask16 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrad512_mask ((__v16si) __A,
       (__v4si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
# 1666 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rcp14_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_rcp14pd512_mask ((__v8df) __A,
         (__v8df)
         _mm512_undefined_pd (),
         (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rcp14_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rcp14pd512_mask ((__v8df) __A,
         (__v8df) __W,
         (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rcp14_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rcp14pd512_mask ((__v8df) __A,
         (__v8df)
         _mm512_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rcp14_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_rcp14ps512_mask ((__v16sf) __A,
        (__v16sf)
        _mm512_undefined_ps (),
        (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rcp14_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rcp14ps512_mask ((__v16sf) __A,
        (__v16sf) __W,
        (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rcp14_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rcp14ps512_mask ((__v16sf) __A,
        (__v16sf)
        _mm512_setzero_ps (),
        (__mmask16) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp14_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rcp14sd ((__v2df) __B,
        (__v2df) __A);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rcp14_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rcp14sd_mask ((__v2df) __B,
      (__v2df) __A,
      (__v2df) __W,
      (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rcp14_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rcp14sd_mask ((__v2df) __B,
      (__v2df) __A,
      (__v2df) _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp14_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rcp14ss ((__v4sf) __B,
       (__v4sf) __A);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rcp14_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rcp14ss_mask ((__v4sf) __B,
      (__v4sf) __A,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rcp14_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rcp14ss_mask ((__v4sf) __B,
      (__v4sf) __A,
      (__v4sf) _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rsqrt14_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_rsqrt14pd512_mask ((__v8df) __A,
           (__v8df)
           _mm512_undefined_pd (),
           (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rsqrt14_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rsqrt14pd512_mask ((__v8df) __A,
           (__v8df) __W,
           (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rsqrt14_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rsqrt14pd512_mask ((__v8df) __A,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rsqrt14_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_rsqrt14ps512_mask ((__v16sf) __A,
          (__v16sf)
          _mm512_undefined_ps (),
          (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rsqrt14_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rsqrt14ps512_mask ((__v16sf) __A,
          (__v16sf) __W,
          (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rsqrt14_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rsqrt14ps512_mask ((__v16sf) __A,
          (__v16sf)
          _mm512_setzero_ps (),
          (__mmask16) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt14_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rsqrt14sd ((__v2df) __B,
          (__v2df) __A);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rsqrt14_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rsqrt14sd_mask ((__v2df) __B,
       (__v2df) __A,
       (__v2df) __W,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rsqrt14_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_rsqrt14sd_mask ((__v2df) __B,
       (__v2df) __A,
       (__v2df) _mm_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt14_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rsqrt14ss ((__v4sf) __B,
         (__v4sf) __A);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rsqrt14_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rsqrt14ss_mask ((__v4sf) __B,
       (__v4sf) __A,
       (__v4sf) __W,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rsqrt14_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_rsqrt14ss_mask ((__v4sf) __B,
      (__v4sf) __A,
      (__v4sf) _mm_setzero_ps (),
      (__mmask8) __U);
}
# 2061 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi8_epi32 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbd512_mask ((__v16qi) __A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi8_epi32 (__m512i __W, __mmask16 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbd512_mask ((__v16qi) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi8_epi32 (__mmask16 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbd512_mask ((__v16qi) __A,
          (__v16si)
          _mm512_setzero_si512 (),
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi8_epi64 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbq512_mask ((__v16qi) __A,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi8_epi64 (__m512i __W, __mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbq512_mask ((__v16qi) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbq512_mask ((__v16qi) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi16_epi32 (__m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwd512_mask ((__v16hi) __A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi16_epi32 (__m512i __W, __mmask16 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwd512_mask ((__v16hi) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi16_epi32 (__mmask16 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwd512_mask ((__v16hi) __A,
          (__v16si)
          _mm512_setzero_si512 (),
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi16_epi64 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwq512_mask ((__v8hi) __A,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi16_epi64 (__m512i __W, __mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwq512_mask ((__v8hi) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovsxwq512_mask ((__v8hi) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi32_epi64 (__m256i __X)
{
  return (__m512i) __builtin_ia32_pmovsxdq512_mask ((__v8si) __X,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_epi64 (__m512i __W, __mmask8 __U, __m256i __X)
{
  return (__m512i) __builtin_ia32_pmovsxdq512_mask ((__v8si) __X,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi32_epi64 (__mmask8 __U, __m256i __X)
{
  return (__m512i) __builtin_ia32_pmovsxdq512_mask ((__v8si) __X,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu8_epi32 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbd512_mask ((__v16qi) __A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu8_epi32 (__m512i __W, __mmask16 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbd512_mask ((__v16qi) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu8_epi32 (__mmask16 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbd512_mask ((__v16qi) __A,
          (__v16si)
          _mm512_setzero_si512 (),
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu8_epi64 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbq512_mask ((__v16qi) __A,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu8_epi64 (__m512i __W, __mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbq512_mask ((__v16qi) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbq512_mask ((__v16qi) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu16_epi32 (__m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwd512_mask ((__v16hi) __A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu16_epi32 (__m512i __W, __mmask16 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwd512_mask ((__v16hi) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu16_epi32 (__mmask16 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwd512_mask ((__v16hi) __A,
          (__v16si)
          _mm512_setzero_si512 (),
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu16_epi64 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwq512_mask ((__v8hi) __A,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu16_epi64 (__m512i __W, __mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwq512_mask ((__v8hi) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m512i) __builtin_ia32_pmovzxwq512_mask ((__v8hi) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu32_epi64 (__m256i __X)
{
  return (__m512i) __builtin_ia32_pmovzxdq512_mask ((__v8si) __X,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu32_epi64 (__m512i __W, __mmask8 __U, __m256i __X)
{
  return (__m512i) __builtin_ia32_pmovzxdq512_mask ((__v8si) __X,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu32_epi64 (__mmask8 __U, __m256i __X)
{
  return (__m512i) __builtin_ia32_pmovzxdq512_mask ((__v8si) __X,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U);
}
# 3858 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_epi64 (__m512i __A)
{
  return (__m512i) __builtin_ia32_pabsq512_mask ((__v8di) __A,
       (__v8di)
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsq512_mask ((__v8di) __A,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_abs_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsq512_mask ((__v8di) __A,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_epi32 (__m512i __A)
{
  return (__m512i) __builtin_ia32_pabsd512_mask ((__v16si) __A,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsd512_mask ((__v16si) __A,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_abs_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsd512_mask ((__v16si) __A,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastss_ps (__m128 __A)
{
  return (__m512) __builtin_ia32_broadcastss512 ((__v4sf) __A,
       (__v16sf)
       _mm512_undefined_ps (),
       (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastss_ps (__m512 __O, __mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastss512 ((__v4sf) __A,
       (__v16sf) __O, __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastss_ps (__mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastss512 ((__v4sf) __A,
       (__v16sf)
       _mm512_setzero_ps (),
       __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastsd_pd (__m128d __A)
{
  return (__m512d) __builtin_ia32_broadcastsd512 ((__v2df) __A,
        (__v8df)
        _mm512_undefined_pd (),
        (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastsd_pd (__m512d __O, __mmask8 __M, __m128d __A)
{
  return (__m512d) __builtin_ia32_broadcastsd512 ((__v2df) __A,
        (__v8df) __O, __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastsd_pd (__mmask8 __M, __m128d __A)
{
  return (__m512d) __builtin_ia32_broadcastsd512 ((__v2df) __A,
        (__v8df)
        _mm512_setzero_pd (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastd_epi32 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastd512 ((__v4si) __A,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastd_epi32 (__m512i __O, __mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastd512 ((__v4si) __A,
        (__v16si) __O, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastd_epi32 (__mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastd512 ((__v4si) __A,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_epi32 (int __A)
{
  return (__m512i) __builtin_ia32_pbroadcastd512_gpr_mask (__A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16)(-1));
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_set1_epi32 (__m512i __O, __mmask16 __M, int __A)
{
  return (__m512i) __builtin_ia32_pbroadcastd512_gpr_mask (__A, (__v16si) __O,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_set1_epi32 (__mmask16 __M, int __A)
{
  return (__m512i)
  __builtin_ia32_pbroadcastd512_gpr_mask (__A,
       (__v16si) _mm512_setzero_si512 (),
       __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastq_epi64 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastq512 ((__v2di) __A,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastq_epi64 (__m512i __O, __mmask8 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastq512 ((__v2di) __A,
        (__v8di) __O, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastq_epi64 (__mmask8 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastq512 ((__v2di) __A,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_set1_epi64 (long long __A)
{
  return (__m512i) __builtin_ia32_pbroadcastq512_gpr_mask (__A,
          (__v8di)
          _mm512_undefined_epi32 (),
          (__mmask8)(-1));
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_set1_epi64 (__m512i __O, __mmask8 __M, long long __A)
{
  return (__m512i) __builtin_ia32_pbroadcastq512_gpr_mask (__A, (__v8di) __O,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_set1_epi64 (__mmask8 __M, long long __A)
{
  return (__m512i)
  __builtin_ia32_pbroadcastq512_gpr_mask (__A,
       (__v8di) _mm512_setzero_si512 (),
       __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_f32x4 (__m128 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x4_512 ((__v4sf) __A,
           (__v16sf)
           _mm512_undefined_ps (),
           (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_f32x4 (__m512 __O, __mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x4_512 ((__v4sf) __A,
           (__v16sf) __O,
           __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_f32x4 (__mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x4_512 ((__v4sf) __A,
           (__v16sf)
           _mm512_setzero_ps (),
           __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_i32x4 (__m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x4_512 ((__v4si) __A,
            (__v16si)
            _mm512_undefined_epi32 (),
            (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_i32x4 (__m512i __O, __mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x4_512 ((__v4si) __A,
            (__v16si) __O,
            __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_i32x4 (__mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x4_512 ((__v4si) __A,
            (__v16si)
            _mm512_setzero_si512 (),
            __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_f64x4 (__m256d __A)
{
  return (__m512d) __builtin_ia32_broadcastf64x4_512 ((__v4df) __A,
            (__v8df)
            _mm512_undefined_pd (),
            (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_f64x4 (__m512d __O, __mmask8 __M, __m256d __A)
{
  return (__m512d) __builtin_ia32_broadcastf64x4_512 ((__v4df) __A,
            (__v8df) __O,
            __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_f64x4 (__mmask8 __M, __m256d __A)
{
  return (__m512d) __builtin_ia32_broadcastf64x4_512 ((__v4df) __A,
            (__v8df)
            _mm512_setzero_pd (),
            __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_i64x4 (__m256i __A)
{
  return (__m512i) __builtin_ia32_broadcasti64x4_512 ((__v4di) __A,
            (__v8di)
            _mm512_undefined_epi32 (),
            (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_i64x4 (__m512i __O, __mmask8 __M, __m256i __A)
{
  return (__m512i) __builtin_ia32_broadcasti64x4_512 ((__v4di) __A,
            (__v8di) __O,
            __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_i64x4 (__mmask8 __M, __m256i __A)
{
  return (__m512i) __builtin_ia32_broadcasti64x4_512 ((__v4di) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            __M);
}
typedef enum
{
  _MM_PERM_AAAA = 0x00, _MM_PERM_AAAB = 0x01, _MM_PERM_AAAC = 0x02,
  _MM_PERM_AAAD = 0x03, _MM_PERM_AABA = 0x04, _MM_PERM_AABB = 0x05,
  _MM_PERM_AABC = 0x06, _MM_PERM_AABD = 0x07, _MM_PERM_AACA = 0x08,
  _MM_PERM_AACB = 0x09, _MM_PERM_AACC = 0x0A, _MM_PERM_AACD = 0x0B,
  _MM_PERM_AADA = 0x0C, _MM_PERM_AADB = 0x0D, _MM_PERM_AADC = 0x0E,
  _MM_PERM_AADD = 0x0F, _MM_PERM_ABAA = 0x10, _MM_PERM_ABAB = 0x11,
  _MM_PERM_ABAC = 0x12, _MM_PERM_ABAD = 0x13, _MM_PERM_ABBA = 0x14,
  _MM_PERM_ABBB = 0x15, _MM_PERM_ABBC = 0x16, _MM_PERM_ABBD = 0x17,
  _MM_PERM_ABCA = 0x18, _MM_PERM_ABCB = 0x19, _MM_PERM_ABCC = 0x1A,
  _MM_PERM_ABCD = 0x1B, _MM_PERM_ABDA = 0x1C, _MM_PERM_ABDB = 0x1D,
  _MM_PERM_ABDC = 0x1E, _MM_PERM_ABDD = 0x1F, _MM_PERM_ACAA = 0x20,
  _MM_PERM_ACAB = 0x21, _MM_PERM_ACAC = 0x22, _MM_PERM_ACAD = 0x23,
  _MM_PERM_ACBA = 0x24, _MM_PERM_ACBB = 0x25, _MM_PERM_ACBC = 0x26,
  _MM_PERM_ACBD = 0x27, _MM_PERM_ACCA = 0x28, _MM_PERM_ACCB = 0x29,
  _MM_PERM_ACCC = 0x2A, _MM_PERM_ACCD = 0x2B, _MM_PERM_ACDA = 0x2C,
  _MM_PERM_ACDB = 0x2D, _MM_PERM_ACDC = 0x2E, _MM_PERM_ACDD = 0x2F,
  _MM_PERM_ADAA = 0x30, _MM_PERM_ADAB = 0x31, _MM_PERM_ADAC = 0x32,
  _MM_PERM_ADAD = 0x33, _MM_PERM_ADBA = 0x34, _MM_PERM_ADBB = 0x35,
  _MM_PERM_ADBC = 0x36, _MM_PERM_ADBD = 0x37, _MM_PERM_ADCA = 0x38,
  _MM_PERM_ADCB = 0x39, _MM_PERM_ADCC = 0x3A, _MM_PERM_ADCD = 0x3B,
  _MM_PERM_ADDA = 0x3C, _MM_PERM_ADDB = 0x3D, _MM_PERM_ADDC = 0x3E,
  _MM_PERM_ADDD = 0x3F, _MM_PERM_BAAA = 0x40, _MM_PERM_BAAB = 0x41,
  _MM_PERM_BAAC = 0x42, _MM_PERM_BAAD = 0x43, _MM_PERM_BABA = 0x44,
  _MM_PERM_BABB = 0x45, _MM_PERM_BABC = 0x46, _MM_PERM_BABD = 0x47,
  _MM_PERM_BACA = 0x48, _MM_PERM_BACB = 0x49, _MM_PERM_BACC = 0x4A,
  _MM_PERM_BACD = 0x4B, _MM_PERM_BADA = 0x4C, _MM_PERM_BADB = 0x4D,
  _MM_PERM_BADC = 0x4E, _MM_PERM_BADD = 0x4F, _MM_PERM_BBAA = 0x50,
  _MM_PERM_BBAB = 0x51, _MM_PERM_BBAC = 0x52, _MM_PERM_BBAD = 0x53,
  _MM_PERM_BBBA = 0x54, _MM_PERM_BBBB = 0x55, _MM_PERM_BBBC = 0x56,
  _MM_PERM_BBBD = 0x57, _MM_PERM_BBCA = 0x58, _MM_PERM_BBCB = 0x59,
  _MM_PERM_BBCC = 0x5A, _MM_PERM_BBCD = 0x5B, _MM_PERM_BBDA = 0x5C,
  _MM_PERM_BBDB = 0x5D, _MM_PERM_BBDC = 0x5E, _MM_PERM_BBDD = 0x5F,
  _MM_PERM_BCAA = 0x60, _MM_PERM_BCAB = 0x61, _MM_PERM_BCAC = 0x62,
  _MM_PERM_BCAD = 0x63, _MM_PERM_BCBA = 0x64, _MM_PERM_BCBB = 0x65,
  _MM_PERM_BCBC = 0x66, _MM_PERM_BCBD = 0x67, _MM_PERM_BCCA = 0x68,
  _MM_PERM_BCCB = 0x69, _MM_PERM_BCCC = 0x6A, _MM_PERM_BCCD = 0x6B,
  _MM_PERM_BCDA = 0x6C, _MM_PERM_BCDB = 0x6D, _MM_PERM_BCDC = 0x6E,
  _MM_PERM_BCDD = 0x6F, _MM_PERM_BDAA = 0x70, _MM_PERM_BDAB = 0x71,
  _MM_PERM_BDAC = 0x72, _MM_PERM_BDAD = 0x73, _MM_PERM_BDBA = 0x74,
  _MM_PERM_BDBB = 0x75, _MM_PERM_BDBC = 0x76, _MM_PERM_BDBD = 0x77,
  _MM_PERM_BDCA = 0x78, _MM_PERM_BDCB = 0x79, _MM_PERM_BDCC = 0x7A,
  _MM_PERM_BDCD = 0x7B, _MM_PERM_BDDA = 0x7C, _MM_PERM_BDDB = 0x7D,
  _MM_PERM_BDDC = 0x7E, _MM_PERM_BDDD = 0x7F, _MM_PERM_CAAA = 0x80,
  _MM_PERM_CAAB = 0x81, _MM_PERM_CAAC = 0x82, _MM_PERM_CAAD = 0x83,
  _MM_PERM_CABA = 0x84, _MM_PERM_CABB = 0x85, _MM_PERM_CABC = 0x86,
  _MM_PERM_CABD = 0x87, _MM_PERM_CACA = 0x88, _MM_PERM_CACB = 0x89,
  _MM_PERM_CACC = 0x8A, _MM_PERM_CACD = 0x8B, _MM_PERM_CADA = 0x8C,
  _MM_PERM_CADB = 0x8D, _MM_PERM_CADC = 0x8E, _MM_PERM_CADD = 0x8F,
  _MM_PERM_CBAA = 0x90, _MM_PERM_CBAB = 0x91, _MM_PERM_CBAC = 0x92,
  _MM_PERM_CBAD = 0x93, _MM_PERM_CBBA = 0x94, _MM_PERM_CBBB = 0x95,
  _MM_PERM_CBBC = 0x96, _MM_PERM_CBBD = 0x97, _MM_PERM_CBCA = 0x98,
  _MM_PERM_CBCB = 0x99, _MM_PERM_CBCC = 0x9A, _MM_PERM_CBCD = 0x9B,
  _MM_PERM_CBDA = 0x9C, _MM_PERM_CBDB = 0x9D, _MM_PERM_CBDC = 0x9E,
  _MM_PERM_CBDD = 0x9F, _MM_PERM_CCAA = 0xA0, _MM_PERM_CCAB = 0xA1,
  _MM_PERM_CCAC = 0xA2, _MM_PERM_CCAD = 0xA3, _MM_PERM_CCBA = 0xA4,
  _MM_PERM_CCBB = 0xA5, _MM_PERM_CCBC = 0xA6, _MM_PERM_CCBD = 0xA7,
  _MM_PERM_CCCA = 0xA8, _MM_PERM_CCCB = 0xA9, _MM_PERM_CCCC = 0xAA,
  _MM_PERM_CCCD = 0xAB, _MM_PERM_CCDA = 0xAC, _MM_PERM_CCDB = 0xAD,
  _MM_PERM_CCDC = 0xAE, _MM_PERM_CCDD = 0xAF, _MM_PERM_CDAA = 0xB0,
  _MM_PERM_CDAB = 0xB1, _MM_PERM_CDAC = 0xB2, _MM_PERM_CDAD = 0xB3,
  _MM_PERM_CDBA = 0xB4, _MM_PERM_CDBB = 0xB5, _MM_PERM_CDBC = 0xB6,
  _MM_PERM_CDBD = 0xB7, _MM_PERM_CDCA = 0xB8, _MM_PERM_CDCB = 0xB9,
  _MM_PERM_CDCC = 0xBA, _MM_PERM_CDCD = 0xBB, _MM_PERM_CDDA = 0xBC,
  _MM_PERM_CDDB = 0xBD, _MM_PERM_CDDC = 0xBE, _MM_PERM_CDDD = 0xBF,
  _MM_PERM_DAAA = 0xC0, _MM_PERM_DAAB = 0xC1, _MM_PERM_DAAC = 0xC2,
  _MM_PERM_DAAD = 0xC3, _MM_PERM_DABA = 0xC4, _MM_PERM_DABB = 0xC5,
  _MM_PERM_DABC = 0xC6, _MM_PERM_DABD = 0xC7, _MM_PERM_DACA = 0xC8,
  _MM_PERM_DACB = 0xC9, _MM_PERM_DACC = 0xCA, _MM_PERM_DACD = 0xCB,
  _MM_PERM_DADA = 0xCC, _MM_PERM_DADB = 0xCD, _MM_PERM_DADC = 0xCE,
  _MM_PERM_DADD = 0xCF, _MM_PERM_DBAA = 0xD0, _MM_PERM_DBAB = 0xD1,
  _MM_PERM_DBAC = 0xD2, _MM_PERM_DBAD = 0xD3, _MM_PERM_DBBA = 0xD4,
  _MM_PERM_DBBB = 0xD5, _MM_PERM_DBBC = 0xD6, _MM_PERM_DBBD = 0xD7,
  _MM_PERM_DBCA = 0xD8, _MM_PERM_DBCB = 0xD9, _MM_PERM_DBCC = 0xDA,
  _MM_PERM_DBCD = 0xDB, _MM_PERM_DBDA = 0xDC, _MM_PERM_DBDB = 0xDD,
  _MM_PERM_DBDC = 0xDE, _MM_PERM_DBDD = 0xDF, _MM_PERM_DCAA = 0xE0,
  _MM_PERM_DCAB = 0xE1, _MM_PERM_DCAC = 0xE2, _MM_PERM_DCAD = 0xE3,
  _MM_PERM_DCBA = 0xE4, _MM_PERM_DCBB = 0xE5, _MM_PERM_DCBC = 0xE6,
  _MM_PERM_DCBD = 0xE7, _MM_PERM_DCCA = 0xE8, _MM_PERM_DCCB = 0xE9,
  _MM_PERM_DCCC = 0xEA, _MM_PERM_DCCD = 0xEB, _MM_PERM_DCDA = 0xEC,
  _MM_PERM_DCDB = 0xED, _MM_PERM_DCDC = 0xEE, _MM_PERM_DCDD = 0xEF,
  _MM_PERM_DDAA = 0xF0, _MM_PERM_DDAB = 0xF1, _MM_PERM_DDAC = 0xF2,
  _MM_PERM_DDAD = 0xF3, _MM_PERM_DDBA = 0xF4, _MM_PERM_DDBB = 0xF5,
  _MM_PERM_DDBC = 0xF6, _MM_PERM_DDBD = 0xF7, _MM_PERM_DDCA = 0xF8,
  _MM_PERM_DDCB = 0xF9, _MM_PERM_DDCC = 0xFA, _MM_PERM_DDCD = 0xFB,
  _MM_PERM_DDDA = 0xFC, _MM_PERM_DDDB = 0xFD, _MM_PERM_DDDC = 0xFE,
  _MM_PERM_DDDD = 0xFF
} _MM_PERM_ENUM;
# 4552 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rolv_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rolv_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rolv_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rorv_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rorv_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rorv_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rolv_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rolv_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W,
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rolv_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prolvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_rorv_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_rorv_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W,
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_rorv_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_prorvq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        (__mmask8) __U);
}
# 5000 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtu32_sd (__m128d __A, unsigned __B)
{
  return (__m128d) __builtin_ia32_cvtusi2sd32 ((__v2df) __A, __B);
}
# 5109 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi32_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovdb512_mask ((__v16si) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_storeu_epi8 (void * __P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovdb512mem_mask ((__v16qi *) __P, (__v16si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_epi8 (__m128i __O, __mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovdb512_mask ((__v16si) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi32_epi8 (__mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovdb512_mask ((__v16si) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi32_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb512_mask ((__v16si) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi32_storeu_epi8 (void * __P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovsdb512mem_mask ((__v16qi *) __P, (__v16si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi32_epi8 (__m128i __O, __mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb512_mask ((__v16si) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi32_epi8 (__mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb512_mask ((__v16si) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi32_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb512_mask ((__v16si) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi32_storeu_epi8 (void * __P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovusdb512mem_mask ((__v16qi *) __P, (__v16si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi32_epi8 (__m128i __O, __mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb512_mask ((__v16si) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi32_epi8 (__mmask16 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb512_mask ((__v16si) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi32_epi16 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovdw512_mask ((__v16si) __A,
        (__v16hi)
        _mm256_undefined_si256 (),
        (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_storeu_epi16 (void * __P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovdw512mem_mask ((__v16hi *) __P, (__v16si) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_epi16 (__m256i __O, __mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovdw512_mask ((__v16si) __A,
        (__v16hi) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi32_epi16 (__mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovdw512_mask ((__v16si) __A,
        (__v16hi)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi32_epi16 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsdw512_mask ((__v16si) __A,
         (__v16hi)
         _mm256_undefined_si256 (),
         (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi32_storeu_epi16 (void *__P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovsdw512mem_mask ((__v16hi*) __P, (__v16si) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi32_epi16 (__m256i __O, __mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsdw512_mask ((__v16si) __A,
         (__v16hi) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi32_epi16 (__mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsdw512_mask ((__v16si) __A,
         (__v16hi)
         _mm256_setzero_si256 (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi32_epi16 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusdw512_mask ((__v16si) __A,
          (__v16hi)
          _mm256_undefined_si256 (),
          (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi32_storeu_epi16 (void *__P, __mmask16 __M, __m512i __A)
{
  __builtin_ia32_pmovusdw512mem_mask ((__v16hi*) __P, (__v16si) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi32_epi16 (__m256i __O, __mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusdw512_mask ((__v16si) __A,
          (__v16hi) __O,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi32_epi16 (__mmask16 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusdw512_mask ((__v16si) __A,
          (__v16hi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi64_epi32 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovqd512_mask ((__v8di) __A,
        (__v8si)
        _mm256_undefined_si256 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_storeu_epi32 (void* __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovqd512mem_mask ((__v8si *) __P, (__v8di) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_epi32 (__m256i __O, __mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovqd512_mask ((__v8di) __A,
        (__v8si) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi64_epi32 (__mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovqd512_mask ((__v8di) __A,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi64_epi32 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsqd512_mask ((__v8di) __A,
         (__v8si)
         _mm256_undefined_si256 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_storeu_epi32 (void *__P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovsqd512mem_mask ((__v8si *) __P, (__v8di) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_epi32 (__m256i __O, __mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsqd512_mask ((__v8di) __A,
         (__v8si) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi64_epi32 (__mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovsqd512_mask ((__v8di) __A,
         (__v8si)
         _mm256_setzero_si256 (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi64_epi32 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusqd512_mask ((__v8di) __A,
          (__v8si)
          _mm256_undefined_si256 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_storeu_epi32 (void* __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovusqd512mem_mask ((__v8si*) __P, (__v8di) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_epi32 (__m256i __O, __mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusqd512_mask ((__v8di) __A,
          (__v8si) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi64_epi32 (__mmask8 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovusqd512_mask ((__v8di) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi64_epi16 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqw512_mask ((__v8di) __A,
        (__v8hi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_storeu_epi16 (void *__P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovqw512mem_mask ((__v8hi *) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_epi16 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqw512_mask ((__v8di) __A,
        (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi64_epi16 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqw512_mask ((__v8di) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi64_epi16 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw512_mask ((__v8di) __A,
         (__v8hi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_storeu_epi16 (void * __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovsqw512mem_mask ((__v8hi *) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_epi16 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw512_mask ((__v8di) __A,
         (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi64_epi16 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw512_mask ((__v8di) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi64_epi16 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw512_mask ((__v8di) __A,
          (__v8hi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_storeu_epi16 (void *__P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovusqw512mem_mask ((__v8hi*) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_epi16 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw512_mask ((__v8di) __A,
          (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi64_epi16 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw512_mask ((__v8di) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi64_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqb512_mask ((__v8di) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_storeu_epi8 (void * __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovqb512mem_mask ((__v16qi *) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_epi8 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqb512_mask ((__v8di) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi64_epi8 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovqb512_mask ((__v8di) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi64_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb512_mask ((__v8di) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_storeu_epi8 (void * __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovsqb512mem_mask ((__v16qi *) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi64_epi8 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb512_mask ((__v8di) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi64_epi8 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb512_mask ((__v8di) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi64_epi8 (__m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb512_mask ((__v8di) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_storeu_epi8 (void * __P, __mmask8 __M, __m512i __A)
{
  __builtin_ia32_pmovusqb512mem_mask ((__v16qi *) __P, (__v8di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi64_epi8 (__m128i __O, __mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb512_mask ((__v8di) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi64_epi8 (__mmask8 __M, __m512i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb512_mask ((__v8di) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi32_pd (__m256i __A)
{
  return (__m512d) __builtin_ia32_cvtdq2pd512_mask ((__v8si) __A,
          (__v8df)
          _mm512_undefined_pd (),
          (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_pd (__m512d __W, __mmask8 __U, __m256i __A)
{
  return (__m512d) __builtin_ia32_cvtdq2pd512_mask ((__v8si) __A,
          (__v8df) __W,
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi32_pd (__mmask8 __U, __m256i __A)
{
  return (__m512d) __builtin_ia32_cvtdq2pd512_mask ((__v8si) __A,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu32_pd (__m256i __A)
{
  return (__m512d) __builtin_ia32_cvtudq2pd512_mask ((__v8si) __A,
           (__v8df)
           _mm512_undefined_pd (),
           (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu32_pd (__m512d __W, __mmask8 __U, __m256i __A)
{
  return (__m512d) __builtin_ia32_cvtudq2pd512_mask ((__v8si) __A,
           (__v8df) __W,
           (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu32_pd (__mmask8 __U, __m256i __A)
{
  return (__m512d) __builtin_ia32_cvtudq2pd512_mask ((__v8si) __A,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) __U);
}
# 6123 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_loadu_pd (void const *__P)
{
  return *(__m512d_u *)__P;
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_pd (__m512d __W, __mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_loadupd512_mask ((const double *) __P,
         (__v8df) __W,
         (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_pd (__mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_loadupd512_mask ((const double *) __P,
         (__v8df)
         _mm512_setzero_pd (),
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_storeu_pd (void *__P, __m512d __A)
{
  *(__m512d_u *)__P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_pd (void *__P, __mmask8 __U, __m512d __A)
{
  __builtin_ia32_storeupd512_mask ((double *) __P, (__v8df) __A,
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_loadu_ps (void const *__P)
{
  return *(__m512_u *)__P;
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_ps (__m512 __W, __mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_loadups512_mask ((const float *) __P,
        (__v16sf) __W,
        (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_ps (__mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_loadups512_mask ((const float *) __P,
        (__v16sf)
        _mm512_setzero_ps (),
        (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_storeu_ps (void *__P, __m512 __A)
{
  *(__m512_u *)__P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_ps (void *__P, __mmask16 __U, __m512 __A)
{
  __builtin_ia32_storeups512_mask ((float *) __P, (__v16sf) __A,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_epi64 (__m512i __W, __mmask8 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddqudi512_mask ((const long long *) __P,
           (__v8di) __W,
           (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddqudi512_mask ((const long long *) __P,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_epi64 (void *__P, __mmask8 __U, __m512i __A)
{
  __builtin_ia32_storedqudi512_mask ((long long *) __P, (__v8di) __A,
         (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_loadu_si512 (void const *__P)
{
  return *(__m512i_u *)__P;
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_epi32 (__m512i __W, __mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddqusi512_mask ((const int *) __P,
           (__v16si) __W,
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_epi32 (__mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddqusi512_mask ((const int *) __P,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_storeu_si512 (void *__P, __m512i __A)
{
  *(__m512i_u *)__P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_epi32 (void *__P, __mmask16 __U, __m512i __A)
{
  __builtin_ia32_storedqusi512_mask ((int *) __P, (__v16si) __A,
         (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutevar_pd (__m512d __A, __m512i __C)
{
  return (__m512d) __builtin_ia32_vpermilvarpd512_mask ((__v8df) __A,
       (__v8di) __C,
       (__v8df)
       _mm512_undefined_pd (),
       (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutevar_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512i __C)
{
  return (__m512d) __builtin_ia32_vpermilvarpd512_mask ((__v8df) __A,
       (__v8di) __C,
       (__v8df) __W,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutevar_pd (__mmask8 __U, __m512d __A, __m512i __C)
{
  return (__m512d) __builtin_ia32_vpermilvarpd512_mask ((__v8df) __A,
       (__v8di) __C,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutevar_ps (__m512 __A, __m512i __C)
{
  return (__m512) __builtin_ia32_vpermilvarps512_mask ((__v16sf) __A,
             (__v16si) __C,
             (__v16sf)
             _mm512_undefined_ps (),
             (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutevar_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512i __C)
{
  return (__m512) __builtin_ia32_vpermilvarps512_mask ((__v16sf) __A,
             (__v16si) __C,
             (__v16sf) __W,
             (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutevar_ps (__mmask16 __U, __m512 __A, __m512i __C)
{
  return (__m512) __builtin_ia32_vpermilvarps512_mask ((__v16sf) __A,
             (__v16si) __C,
             (__v16sf)
             _mm512_setzero_ps (),
             (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_epi64 (__m512i __A, __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varq512_mask ((__v8di) __I
                       ,
             (__v8di) __A,
             (__v8di) __B,
             (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_epi64 (__m512i __A, __mmask8 __U, __m512i __I,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varq512_mask ((__v8di) __I
                       ,
             (__v8di) __A,
             (__v8di) __B,
             (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_epi64 (__m512i __A, __m512i __I,
     __mmask8 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermi2varq512_mask ((__v8di) __A,
             (__v8di) __I
                       ,
             (__v8di) __B,
             (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_epi64 (__mmask8 __U, __m512i __A,
     __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varq512_maskz ((__v8di) __I
                 ,
       (__v8di) __A,
       (__v8di) __B,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_epi32 (__m512i __A, __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2vard512_mask ((__v16si) __I
                       ,
             (__v16si) __A,
             (__v16si) __B,
             (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_epi32 (__m512i __A, __mmask16 __U,
    __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2vard512_mask ((__v16si) __I
                       ,
             (__v16si) __A,
             (__v16si) __B,
             (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_epi32 (__m512i __A, __m512i __I,
     __mmask16 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermi2vard512_mask ((__v16si) __A,
             (__v16si) __I
                       ,
             (__v16si) __B,
             (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_epi32 (__mmask16 __U, __m512i __A,
     __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2vard512_maskz ((__v16si) __I
                 ,
       (__v16si) __A,
       (__v16si) __B,
       (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_pd (__m512d __A, __m512i __I, __m512d __B)
{
  return (__m512d) __builtin_ia32_vpermt2varpd512_mask ((__v8di) __I
                 ,
       (__v8df) __A,
       (__v8df) __B,
       (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_pd (__m512d __A, __mmask8 __U, __m512i __I,
        __m512d __B)
{
  return (__m512d) __builtin_ia32_vpermt2varpd512_mask ((__v8di) __I
                 ,
       (__v8df) __A,
       (__v8df) __B,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_pd (__m512d __A, __m512i __I, __mmask8 __U,
         __m512d __B)
{
  return (__m512d) __builtin_ia32_vpermi2varpd512_mask ((__v8df) __A,
       (__v8di) __I
                 ,
       (__v8df) __B,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_pd (__mmask8 __U, __m512d __A, __m512i __I,
         __m512d __B)
{
  return (__m512d) __builtin_ia32_vpermt2varpd512_maskz ((__v8di) __I
                  ,
        (__v8df) __A,
        (__v8df) __B,
        (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_ps (__m512 __A, __m512i __I, __m512 __B)
{
  return (__m512) __builtin_ia32_vpermt2varps512_mask ((__v16si) __I
                       ,
             (__v16sf) __A,
             (__v16sf) __B,
             (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_ps (__m512 __A, __mmask16 __U, __m512i __I, __m512 __B)
{
  return (__m512) __builtin_ia32_vpermt2varps512_mask ((__v16si) __I
                       ,
             (__v16sf) __A,
             (__v16sf) __B,
             (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_ps (__m512 __A, __m512i __I, __mmask16 __U,
         __m512 __B)
{
  return (__m512) __builtin_ia32_vpermi2varps512_mask ((__v16sf) __A,
             (__v16si) __I
                       ,
             (__v16sf) __B,
             (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_ps (__mmask16 __U, __m512 __A, __m512i __I,
         __m512 __B)
{
  return (__m512) __builtin_ia32_vpermt2varps512_maskz ((__v16si) __I
                 ,
       (__v16sf) __A,
       (__v16sf) __B,
       (__mmask16) __U);
}
# 6709 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_epi64 (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvardi512_mask ((__v8di) __Y,
           (__v8di) __X,
           (__v8di)
           _mm512_setzero_si512 (),
           __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_epi64 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvardi512_mask ((__v8di) __Y,
           (__v8di) __X,
           (__v8di)
           _mm512_undefined_epi32 (),
           (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_epi64 (__m512i __W, __mmask8 __M, __m512i __X,
          __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvardi512_mask ((__v8di) __Y,
           (__v8di) __X,
           (__v8di) __W,
           __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_epi32 (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvarsi512_mask ((__v16si) __Y,
           (__v16si) __X,
           (__v16si)
           _mm512_setzero_si512 (),
           __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_epi32 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvarsi512_mask ((__v16si) __Y,
           (__v16si) __X,
           (__v16si)
           _mm512_undefined_epi32 (),
           (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_epi32 (__m512i __W, __mmask16 __M, __m512i __X,
          __m512i __Y)
{
  return (__m512i) __builtin_ia32_permvarsi512_mask ((__v16si) __Y,
           (__v16si) __X,
           (__v16si) __W,
           __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_pd (__m512i __X, __m512d __Y)
{
  return (__m512d) __builtin_ia32_permvardf512_mask ((__v8df) __Y,
           (__v8di) __X,
           (__v8df)
           _mm512_undefined_pd (),
           (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_pd (__m512d __W, __mmask8 __U, __m512i __X, __m512d __Y)
{
  return (__m512d) __builtin_ia32_permvardf512_mask ((__v8df) __Y,
           (__v8di) __X,
           (__v8df) __W,
           (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_pd (__mmask8 __U, __m512i __X, __m512d __Y)
{
  return (__m512d) __builtin_ia32_permvardf512_mask ((__v8df) __Y,
           (__v8di) __X,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_ps (__m512i __X, __m512 __Y)
{
  return (__m512) __builtin_ia32_permvarsf512_mask ((__v16sf) __Y,
          (__v16si) __X,
          (__v16sf)
          _mm512_undefined_ps (),
          (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_ps (__m512 __W, __mmask16 __U, __m512i __X, __m512 __Y)
{
  return (__m512) __builtin_ia32_permvarsf512_mask ((__v16sf) __Y,
          (__v16si) __X,
          (__v16sf) __W,
          (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_ps (__mmask16 __U, __m512i __X, __m512 __Y)
{
  return (__m512) __builtin_ia32_permvarsf512_mask ((__v16sf) __Y,
          (__v16si) __X,
          (__v16sf)
          _mm512_setzero_ps (),
          (__mmask16) __U);
}
# 7144 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movehdup_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_movshdup512_mask ((__v16sf) __A,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_movehdup_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movshdup512_mask ((__v16sf) __A,
         (__v16sf) __W,
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_movehdup_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movshdup512_mask ((__v16sf) __A,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_moveldup_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_movsldup512_mask ((__v16sf) __A,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_moveldup_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movsldup512_mask ((__v16sf) __A,
         (__v16sf) __W,
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_moveldup_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_movsldup512_mask ((__v16sf) __A,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_or_si512 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A | (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_or_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A | (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_or_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pord512_mask ((__v16si) __A,
      (__v16si) __B,
      (__v16si) __W,
      (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_or_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pord512_mask ((__v16si) __A,
      (__v16si) __B,
      (__v16si)
      _mm512_setzero_si512 (),
      (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_or_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A | (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_or_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_porq512_mask ((__v8di) __A,
      (__v8di) __B,
      (__v8di) __W,
      (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_or_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_porq512_mask ((__v8di) __A,
      (__v8di) __B,
      (__v8di)
      _mm512_setzero_si512 (),
      (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_xor_si512 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A ^ (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_xor_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A ^ (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_xor_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pxord512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_xor_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pxord512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_xor_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A ^ (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_xor_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pxorq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_xor_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pxorq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
# 7509 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_and_si512 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A & (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_and_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v16su) __A & (__v16su) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_and_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_and_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandd512_mask ((__v16si) __A,
       (__v16si) __B,
       (__v16si)
       _mm512_setzero_si512 (),
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_and_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A & (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_and_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di) __W, __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_and_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandq512_mask ((__v8di) __A,
       (__v8di) __B,
       (__v8di)
       _mm512_setzero_pd (),
       __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_andnot_si512 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_andnot_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_andnot_epi32 (__m512i __W, __mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_andnot_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_andnot_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_andnot_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W, __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_andnot_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pandnq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_pd (),
        __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_test_epi32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ptestmd512 ((__v16si) __A,
      (__v16si) __B,
      (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_test_epi32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ptestmd512 ((__v16si) __A,
      (__v16si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_test_epi64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq512 ((__v8di) __A,
            (__v8di) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_test_epi64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq512 ((__v8di) __A, (__v8di) __B, __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_testn_epi32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmd512 ((__v16si) __A,
       (__v16si) __B,
       (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_testn_epi32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmd512 ((__v16si) __A,
       (__v16si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_testn_epi64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq512 ((__v8di) __A,
      (__v8di) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_testn_epi64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq512 ((__v8di) __A,
      (__v8di) __B, __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_ps (__m512 __A)
{
  return (__m512) _mm512_and_epi32 ((__m512i) __A,
        _mm512_set1_epi32 (0x7fffffff));
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) _mm512_mask_and_epi32 ((__m512i) __W, __U, (__m512i) __A,
      _mm512_set1_epi32 (0x7fffffff));
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_pd (__m512d __A)
{
  return (__m512d) _mm512_and_epi64 ((__m512i) __A,
         _mm512_set1_epi64 (0x7fffffffffffffffLL));
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d)
  _mm512_mask_and_epi64 ((__m512i) __W, __U, (__m512i) __A,
    _mm512_set1_epi64 (0x7fffffffffffffffLL));
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhdq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si)
           _mm512_undefined_epi32 (),
           (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_epi32 (__m512i __W, __mmask16 __U, __m512i __A,
       __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhdq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si) __W,
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhdq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di)
            _mm512_undefined_epi32 (),
            (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di) __W,
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckldq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si)
           _mm512_undefined_epi32 (),
           (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_epi32 (__m512i __W, __mmask16 __U, __m512i __A,
       __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckldq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si) __W,
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_epi32 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckldq512_mask ((__v16si) __A,
           (__v16si) __B,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di)
            _mm512_undefined_epi32 (),
            (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_epi64 (__m512i __W, __mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di) __W,
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklqdq512_mask ((__v8di) __A,
            (__v8di) __B,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U);
}
# 8127 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movedup_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_movddup512_mask ((__v8df) __A,
         (__v8df)
         _mm512_undefined_pd (),
         (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_movedup_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_movddup512_mask ((__v8df) __A,
         (__v8df) __W,
         (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_movedup_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_movddup512_mask ((__v8df) __A,
         (__v8df)
         _mm512_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpcklpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_undefined_pd (),
          (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpcklpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df) __W,
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpcklpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpckhpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_undefined_pd (),
          (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpckhpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df) __W,
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_unpckhpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpckhps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpckhps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __W,
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpckhps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U);
}
# 8481 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_stream_si512 (__m512i * __P, __m512i __A)
{
  __builtin_ia32_movntdq512 ((__v8di *) __P, (__v8di) __A);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_stream_ps (float *__P, __m512 __A)
{
  __builtin_ia32_movntps512 (__P, (__v16sf) __A);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_stream_pd (double *__P, __m512d __A)
{
  __builtin_ia32_movntpd512 (__P, (__v8df) __A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_stream_load_si512 (void *__P)
{
  return __builtin_ia32_movntdqa512 ((__v8di *)__P);
}
typedef enum
{
  _MM_MANT_NORM_1_2,
  _MM_MANT_NORM_p5_2,
  _MM_MANT_NORM_p5_1,
  _MM_MANT_NORM_p75_1p5
} _MM_MANTISSA_NORM_ENUM;
typedef enum
{
  _MM_MANT_SIGN_src,
  _MM_MANT_SIGN_zero,
  _MM_MANT_SIGN_nan
} _MM_MANTISSA_SIGN_ENUM;
# 9046 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_floor_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_rndscaleps_mask ((__v16sf) __A,
        (0x01 | 0x00),
        (__v16sf) __A, -1,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_floor_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_rndscalepd_mask ((__v8df) __A,
         (0x01 | 0x00),
         (__v8df) __A, -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_ceil_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_rndscaleps_mask ((__v16sf) __A,
        (0x02 | 0x00),
        (__v16sf) __A, -1,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_ceil_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_rndscalepd_mask ((__v8df) __A,
         (0x02 | 0x00),
         (__v8df) __A, -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_floor_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rndscaleps_mask ((__v16sf) __A,
        (0x01 | 0x00),
        (__v16sf) __W, __U,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_floor_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rndscalepd_mask ((__v8df) __A,
         (0x01 | 0x00),
         (__v8df) __W, __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_ceil_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_rndscaleps_mask ((__v16sf) __A,
        (0x02 | 0x00),
        (__v16sf) __W, __U,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_ceil_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_rndscalepd_mask ((__v8df) __A,
         (0x02 | 0x00),
         (__v8df) __W, __U,
         0x04);
}
# 9225 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epi32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqd512_mask ((__v16si) __A,
           (__v16si) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epi32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqd512_mask ((__v16si) __A,
           (__v16si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epi64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq512_mask ((__v8di) __A,
          (__v8di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epi64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq512_mask ((__v8di) __A,
          (__v8di) __B,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epi32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtd512_mask ((__v16si) __A,
           (__v16si) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epi32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtd512_mask ((__v16si) __A,
           (__v16si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epi64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq512_mask ((__v8di) __A,
          (__v8di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epi64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq512_mask ((__v8di) __A,
          (__v8di) __B,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epi32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 5,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epi32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 5,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epu32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 5,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epu32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 5,
          (__mmask16) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epi64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 5,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epi64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 5,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epu64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 5,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epu64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 5,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epi32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 2,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epi32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 2,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epu32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 2,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epu32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 2,
          (__mmask16) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epi64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 2,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epi64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 2,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epu64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 2,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epu64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 2,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epi32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 1,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epi32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 1,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epu32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 1,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epu32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 1,
          (__mmask16) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epi64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 1,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epi64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 1,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epu64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 1,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epu64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 1,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epi32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 4,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epi32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_cmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 4,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epu32_mask (__mmask16 __M, __m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 4,
          (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epu32_mask (__m512i __X, __m512i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __X,
          (__v16si) __Y, 4,
          (__mmask16) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epi64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 4,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epi64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 4,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epu64_mask (__mmask8 __M, __m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 4,
          (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epu64_mask (__m512i __X, __m512i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __X,
          (__v8di) __Y, 4,
          (__mmask8) -1);
}
# 10363 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_compressdf512_mask ((__v8df) __A,
            (__v8df) __W,
            (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_compressdf512_mask ((__v8df) __A,
            (__v8df)
            _mm512_setzero_pd (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_pd (void *__P, __mmask8 __U, __m512d __A)
{
  __builtin_ia32_compressstoredf512_mask ((__v8df *) __P, (__v8df) __A,
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_compresssf512_mask ((__v16sf) __A,
           (__v16sf) __W,
           (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_compresssf512_mask ((__v16sf) __A,
           (__v16sf)
           _mm512_setzero_ps (),
           (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_ps (void *__P, __mmask16 __U, __m512 __A)
{
  __builtin_ia32_compressstoresf512_mask ((__v16sf *) __P, (__v16sf) __A,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_compressdi512_mask ((__v8di) __A,
            (__v8di) __W,
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_compressdi512_mask ((__v8di) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_epi64 (void *__P, __mmask8 __U, __m512i __A)
{
  __builtin_ia32_compressstoredi512_mask ((__v8di *) __P, (__v8di) __A,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_compresssi512_mask ((__v16si) __A,
            (__v16si) __W,
            (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_compresssi512_mask ((__v16si) __A,
            (__v16si)
            _mm512_setzero_si512 (),
            (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_epi32 (void *__P, __mmask16 __U, __m512i __A)
{
  __builtin_ia32_compressstoresi512_mask ((__v16si *) __P, (__v16si) __A,
       (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_expanddf512_mask ((__v8df) __A,
          (__v8df) __W,
          (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_expanddf512_maskz ((__v8df) __A,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_pd (__m512d __W, __mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_expandloaddf512_mask ((const __v8df *) __P,
       (__v8df) __W,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_pd (__mmask8 __U, void const *__P)
{
  return (__m512d) __builtin_ia32_expandloaddf512_maskz ((const __v8df *) __P,
        (__v8df)
        _mm512_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_expandsf512_mask ((__v16sf) __A,
         (__v16sf) __W,
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_expandsf512_maskz ((__v16sf) __A,
          (__v16sf)
          _mm512_setzero_ps (),
          (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_ps (__m512 __W, __mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_expandloadsf512_mask ((const __v16sf *) __P,
             (__v16sf) __W,
             (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_ps (__mmask16 __U, void const *__P)
{
  return (__m512) __builtin_ia32_expandloadsf512_maskz ((const __v16sf *) __P,
       (__v16sf)
       _mm512_setzero_ps (),
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_expanddi512_mask ((__v8di) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_expanddi512_maskz ((__v8di) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_epi64 (__m512i __W, __mmask8 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_expandloaddi512_mask ((const __v8di *) __P,
       (__v8di) __W,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m512i)
  __builtin_ia32_expandloaddi512_maskz ((const __v8di *) __P,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_expandsi512_mask ((__v16si) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_expandsi512_maskz ((__v16si) __A,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_epi32 (__m512i __W, __mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_expandloadsi512_mask ((const __v16si *) __P,
       (__v16si) __W,
       (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_epi32 (__mmask16 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_expandloadsi512_maskz ((const __v16si *) __P,
        (__v16si)
        _mm512_setzero_si512
        (), (__mmask16) __U);
}
# 10632 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortest_mask16_u8 (__mmask16 __A, __mmask16 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_kortestchi (__A, __B);
  return (unsigned char) __builtin_ia32_kortestzhi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestz_mask16_u8 (__mmask16 __A, __mmask16 __B)
{
  return (unsigned char) __builtin_ia32_kortestzhi ((__mmask16) __A,
          (__mmask16) __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestc_mask16_u8 (__mmask16 __A, __mmask16 __B)
{
  return (unsigned char) __builtin_ia32_kortestchi ((__mmask16) __A,
          (__mmask16) __B);
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtmask16_u32 (__mmask16 __A)
{
  return (unsigned int) __builtin_ia32_kmovw ((__mmask16 ) __A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtu32_mask16 (unsigned int __A)
{
  return (__mmask16) __builtin_ia32_kmovw ((__mmask16 ) __A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_load_mask16 (__mmask16 *__A)
{
  return (__mmask16) __builtin_ia32_kmovw (*(__mmask16 *) __A);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_store_mask16 (__mmask16 *__A, __mmask16 __B)
{
  *(__mmask16 *) __A = __builtin_ia32_kmovw (__B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kand (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kandhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kandn (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kandnhi ((__mmask16) __A,
          (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kor (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_korhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kortestz (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kortestzhi ((__mmask16) __A,
      (__mmask16) __B);
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kortestc (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kortestchi ((__mmask16) __A,
      (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kxnor (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kxnorhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kxor (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kxorhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_knot (__mmask16 __A)
{
  return (__mmask16) __builtin_ia32_knothi ((__mmask16) __A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kunpackb (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kunpckhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kunpackb_mask16 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask16) __builtin_ia32_kunpckhi ((__mmask16) __A, (__mmask16) __B);
}
# 10827 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epi64 (__mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epi64 (__m512i __W, __mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epi64 (__m512i __W, __mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epi64 (__mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epu64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epu64 (__mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epu64 (__m512i __W, __mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epu64 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_undefined_epi32 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epu64 (__m512i __W, __mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epu64 (__mmask8 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epi32 (__mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epi32 (__m512i __W, __mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epi32 (__mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epi32 (__m512i __W, __mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsd512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epu32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epu32 (__mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epu32 (__m512i __W, __mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epu32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_undefined_epi32 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epu32 (__mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si)
        _mm512_setzero_si512 (),
        __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epu32 (__m512i __W, __mmask16 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminud512_mask ((__v16si) __A,
        (__v16si) __B,
        (__v16si) __W, __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpcklps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpcklps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __W,
         (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_unpcklps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U);
}
# 11275 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_blend_pd (__mmask8 __U, __m512d __A, __m512d __W)
{
  return (__m512d) __builtin_ia32_blendmpd_512_mask ((__v8df) __A,
           (__v8df) __W,
           (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_blend_ps (__mmask16 __U, __m512 __A, __m512 __W)
{
  return (__m512) __builtin_ia32_blendmps_512_mask ((__v16sf) __A,
          (__v16sf) __W,
          (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_blend_epi64 (__mmask8 __U, __m512i __A, __m512i __W)
{
  return (__m512i) __builtin_ia32_blendmq_512_mask ((__v8di) __A,
          (__v8di) __W,
          (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_blend_epi32 (__mmask16 __U, __m512i __A, __m512i __W)
{
  return (__m512i) __builtin_ia32_blendmd_512_mask ((__v16si) __A,
          (__v16si) __W,
          (__mmask16) __U);
}
# 11438 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sqrt_pd (__m512d __A)
{
  return (__m512d) __builtin_ia32_sqrtpd512_mask ((__v8df) __A,
        (__v8df)
        _mm512_undefined_pd (),
        (__mmask8) -1,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sqrt_pd (__m512d __W, __mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_sqrtpd512_mask ((__v8df) __A,
        (__v8df) __W,
        (__mmask8) __U,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sqrt_pd (__mmask8 __U, __m512d __A)
{
  return (__m512d) __builtin_ia32_sqrtpd512_mask ((__v8df) __A,
        (__v8df)
        _mm512_setzero_pd (),
        (__mmask8) __U,
        0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sqrt_ps (__m512 __A)
{
  return (__m512) __builtin_ia32_sqrtps512_mask ((__v16sf) __A,
       (__v16sf)
       _mm512_undefined_ps (),
       (__mmask16) -1,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sqrt_ps (__m512 __W, __mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_sqrtps512_mask ((__v16sf) __A,
       (__v16sf) __W,
       (__mmask16) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sqrt_ps (__mmask16 __U, __m512 __A)
{
  return (__m512) __builtin_ia32_sqrtps512_mask ((__v16sf) __A,
       (__v16sf)
       _mm512_setzero_ps (),
       (__mmask16) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_pd (__m512d __A, __m512d __B)
{
  return (__m512d) ((__v8df)__A + (__v8df)__B);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_addpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_addpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_ps (__m512 __A, __m512 __B)
{
  return (__m512) ((__v16sf)__A + (__v16sf)__B);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_addps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_addps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_addsd_mask_round ((__v2df) __A,
      (__v2df) __B,
      (__v2df) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_addsd_mask_round ((__v2df) __A,
      (__v2df) __B,
      (__v2df)
      _mm_setzero_pd (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_addss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_addss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_pd (__m512d __A, __m512d __B)
{
  return (__m512d) ((__v8df)__A - (__v8df)__B);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_subpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_subpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_ps (__m512 __A, __m512 __B)
{
  return (__m512) ((__v16sf)__A - (__v16sf)__B);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_subps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_subps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_subsd_mask_round ((__v2df) __A,
      (__v2df) __B,
      (__v2df) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_subsd_mask_round ((__v2df) __A,
      (__v2df) __B,
      (__v2df)
      _mm_setzero_pd (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_subss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_subss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mul_pd (__m512d __A, __m512d __B)
{
  return (__m512d) ((__v8df)__A * (__v8df)__B);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mul_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_mulpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mul_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_mulpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mul_ps (__m512 __A, __m512 __B)
{
  return (__m512) ((__v16sf)__A * (__v16sf)__B);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mul_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_mulps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mul_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_mulps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_sd (__m128d __W, __mmask8 __U, __m128d __A,
     __m128d __B)
{
  return (__m128d) __builtin_ia32_mulsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U,
        0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_mulsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U,
        0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_ss (__m128 __W, __mmask8 __U, __m128 __A,
     __m128 __B)
{
  return (__m128) __builtin_ia32_mulss_mask_round ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf) __W,
       (__mmask8) __U,
        0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_mulss_mask_round ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_div_pd (__m512d __M, __m512d __V)
{
  return (__m512d) ((__v8df)__M / (__v8df)__V);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_div_pd (__m512d __W, __mmask8 __U, __m512d __M, __m512d __V)
{
  return (__m512d) __builtin_ia32_divpd512_mask ((__v8df) __M,
       (__v8df) __V,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_div_pd (__mmask8 __U, __m512d __M, __m512d __V)
{
  return (__m512d) __builtin_ia32_divpd512_mask ((__v8df) __M,
       (__v8df) __V,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_div_ps (__m512 __A, __m512 __B)
{
  return (__m512) ((__v16sf)__A / (__v16sf)__B);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_div_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_divps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_div_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_divps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_div_sd (__m128d __W, __mmask8 __U, __m128d __A,
     __m128d __B)
{
  return (__m128d) __builtin_ia32_divsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U,
        0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_div_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_divsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U,
        0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_div_ss (__m128 __W, __mmask8 __U, __m128 __A,
     __m128 __B)
{
  return (__m128) __builtin_ia32_divss_mask_round ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf) __W,
       (__mmask8) __U,
        0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_div_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_divss_mask_round ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U,
        0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_maxpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_undefined_pd (),
       (__mmask8) -1,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_maxpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_maxpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_maxps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_undefined_ps (),
      (__mmask16) -1,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_maxps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_maxps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_maxsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_maxsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_minpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_undefined_pd (),
       (__mmask8) -1,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_minpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_minpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_minps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_undefined_ps (),
      (__mmask16) -1,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_minps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U,
      0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_minps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U,
      0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_sd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_minsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U,
       0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_sd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_minsd_mask_round ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U,
       0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_ss (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U,
      0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_ss (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minss_mask_round ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U,
      0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_scalef_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_scalefpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_undefined_pd (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_scalef_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_scalefpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_scalef_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_scalefpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_scalef_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_scalefps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_scalef_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_scalefps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __W,
         (__mmask16) __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_scalef_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_scalefps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U,
         0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_scalef_sd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_scalefsd_mask_round ((__v2df) __A,
          (__v2df) __B,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_scalef_ss (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_scalefss_mask_round ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) -1,
         0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmadd_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df) __C,
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmadd_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask ((__v8df) __A,
          (__v8df) __B,
          (__v8df) __C,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmadd_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask3 ((__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmadd_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_maskz ((__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmadd_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __C,
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmadd_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __C,
         (__mmask16) __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmadd_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask3 ((__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmadd_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_maskz ((__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmsub_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask ((__v8df) __A,
          (__v8df) __B,
          -(__v8df) __C,
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmsub_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask ((__v8df) __A,
          (__v8df) __B,
          -(__v8df) __C,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmsub_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfmsubpd512_mask3 ((__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmsub_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_maskz ((__v8df) __A,
           (__v8df) __B,
           -(__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmsub_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         -(__v16sf) __C,
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmsub_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask ((__v16sf) __A,
         (__v16sf) __B,
         -(__v16sf) __C,
         (__mmask16) __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmsub_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfmsubps512_mask3 ((__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmsub_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_maskz ((__v16sf) __A,
          (__v16sf) __B,
          -(__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmaddsub_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_mask ((__v8df) __A,
             (__v8df) __B,
             (__v8df) __C,
             (__mmask8) -1,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmaddsub_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_mask ((__v8df) __A,
             (__v8df) __B,
             (__v8df) __C,
             (__mmask8) __U,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmaddsub_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_mask3 ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __C,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmaddsub_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_maskz ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __C,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmaddsub_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            (__v16sf) __C,
            (__mmask16) -1,
            0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmaddsub_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            (__v16sf) __C,
            (__mmask16) __U,
            0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmaddsub_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_mask3 ((__v16sf) __A,
             (__v16sf) __B,
             (__v16sf) __C,
             (__mmask16) __U,
             0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmaddsub_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_maskz ((__v16sf) __A,
             (__v16sf) __B,
             (__v16sf) __C,
             (__mmask16) __U,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmsubadd_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_mask ((__v8df) __A,
             (__v8df) __B,
             -(__v8df) __C,
             (__mmask8) -1,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmsubadd_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_mask ((__v8df) __A,
             (__v8df) __B,
             -(__v8df) __C,
             (__mmask8) __U,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmsubadd_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfmsubaddpd512_mask3 ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __C,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmsubadd_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddsubpd512_maskz ((__v8df) __A,
       (__v8df) __B,
       -(__v8df) __C,
       (__mmask8) __U,
       0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fmsubadd_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            -(__v16sf) __C,
            (__mmask16) -1,
            0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fmsubadd_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            -(__v16sf) __C,
            (__mmask16) __U,
            0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fmsubadd_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfmsubaddps512_mask3 ((__v16sf) __A,
             (__v16sf) __B,
             (__v16sf) __C,
             (__mmask16) __U,
             0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fmsubadd_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddsubps512_maskz ((__v16sf) __A,
             (__v16sf) __B,
             -(__v16sf) __C,
             (__mmask16) __U,
             0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fnmadd_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask (-(__v8df) __A,
          (__v8df) __B,
          (__v8df) __C,
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fnmadd_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfnmaddpd512_mask ((__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fnmadd_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask3 (-(__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fnmadd_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_maskz (-(__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fnmadd_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask (-(__v16sf) __A,
         (__v16sf) __B,
         (__v16sf) __C,
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fnmadd_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfnmaddps512_mask ((__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fnmadd_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask3 (-(__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fnmadd_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_maskz (-(__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fnmsub_pd (__m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_mask (-(__v8df) __A,
          (__v8df) __B,
          -(__v8df) __C,
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fnmsub_pd (__m512d __A, __mmask8 __U, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfnmsubpd512_mask ((__v8df) __A,
           (__v8df) __B,
           (__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fnmsub_pd (__m512d __A, __m512d __B, __m512d __C, __mmask8 __U)
{
  return (__m512d) __builtin_ia32_vfnmsubpd512_mask3 ((__v8df) __A,
            (__v8df) __B,
            (__v8df) __C,
            (__mmask8) __U,
            0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fnmsub_pd (__mmask8 __U, __m512d __A, __m512d __B, __m512d __C)
{
  return (__m512d) __builtin_ia32_vfmaddpd512_maskz (-(__v8df) __A,
           (__v8df) __B,
           -(__v8df) __C,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_fnmsub_ps (__m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_mask (-(__v16sf) __A,
         (__v16sf) __B,
         -(__v16sf) __C,
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_fnmsub_ps (__m512 __A, __mmask16 __U, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfnmsubps512_mask ((__v16sf) __A,
          (__v16sf) __B,
          (__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask3_fnmsub_ps (__m512 __A, __m512 __B, __m512 __C, __mmask16 __U)
{
  return (__m512) __builtin_ia32_vfnmsubps512_mask3 ((__v16sf) __A,
           (__v16sf) __B,
           (__v16sf) __C,
           (__mmask16) __U,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_fnmsub_ps (__mmask16 __U, __m512 __A, __m512 __B, __m512 __C)
{
  return (__m512) __builtin_ia32_vfmaddps512_maskz (-(__v16sf) __A,
          (__v16sf) __B,
          -(__v16sf) __C,
          (__mmask16) __U,
          0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttpd_epi32 (__m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2dq512_mask ((__v8df) __A,
           (__v8si)
           _mm256_undefined_si256 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttpd_epi32 (__m256i __W, __mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2dq512_mask ((__v8df) __A,
           (__v8si) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttpd_epi32 (__mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2dq512_mask ((__v8df) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttpd_epu32 (__m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2udq512_mask ((__v8df) __A,
            (__v8si)
            _mm256_undefined_si256 (),
            (__mmask8) -1,
            0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttpd_epu32 (__m256i __W, __mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2udq512_mask ((__v8df) __A,
            (__v8si) __W,
            (__mmask8) __U,
            0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttpd_epu32 (__mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2udq512_mask ((__v8df) __A,
            (__v8si)
            _mm256_setzero_si256 (),
            (__mmask8) __U,
            0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtpd_epi32 (__m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2dq512_mask ((__v8df) __A,
          (__v8si)
          _mm256_undefined_si256 (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtpd_epi32 (__m256i __W, __mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2dq512_mask ((__v8df) __A,
          (__v8si) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtpd_epi32 (__mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2dq512_mask ((__v8df) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtpd_epu32 (__m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2udq512_mask ((__v8df) __A,
           (__v8si)
           _mm256_undefined_si256 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtpd_epu32 (__m256i __W, __mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2udq512_mask ((__v8df) __A,
           (__v8si) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtpd_epu32 (__mmask8 __U, __m512d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2udq512_mask ((__v8df) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttps_epi32 (__m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2dq512_mask ((__v16sf) __A,
           (__v16si)
           _mm512_undefined_epi32 (),
           (__mmask16) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttps_epi32 (__m512i __W, __mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2dq512_mask ((__v16sf) __A,
           (__v16si) __W,
           (__mmask16) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttps_epi32 (__mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2dq512_mask ((__v16sf) __A,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttps_epu32 (__m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2udq512_mask ((__v16sf) __A,
            (__v16si)
            _mm512_undefined_epi32 (),
            (__mmask16) -1,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttps_epu32 (__m512i __W, __mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2udq512_mask ((__v16sf) __A,
            (__v16si) __W,
            (__mmask16) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttps_epu32 (__mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvttps2udq512_mask ((__v16sf) __A,
            (__v16si)
            _mm512_setzero_si512 (),
            (__mmask16) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtps_epi32 (__m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2dq512_mask ((__v16sf) __A,
          (__v16si)
          _mm512_undefined_epi32 (),
          (__mmask16) -1,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtps_epi32 (__m512i __W, __mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2dq512_mask ((__v16sf) __A,
          (__v16si) __W,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtps_epi32 (__mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2dq512_mask ((__v16sf) __A,
          (__v16si)
          _mm512_setzero_si512 (),
          (__mmask16) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtps_epu32 (__m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2udq512_mask ((__v16sf) __A,
           (__v16si)
           _mm512_undefined_epi32 (),
           (__mmask16) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtps_epu32 (__m512i __W, __mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2udq512_mask ((__v16sf) __A,
           (__v16si) __W,
           (__mmask16) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtps_epu32 (__mmask16 __U, __m512 __A)
{
  return (__m512i) __builtin_ia32_cvtps2udq512_mask ((__v16sf) __A,
           (__v16si)
           _mm512_setzero_si512 (),
           (__mmask16) __U,
           0x04);
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsd_f64 (__m512d __A)
{
  return __A[0];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtss_f32 (__m512 __A)
{
  return __A[0];
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtu64_ss (__m128 __A, unsigned long long __B)
{
  return (__m128) __builtin_ia32_cvtusi2ss64 ((__v4sf) __A, __B,
           0x04);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtu64_sd (__m128d __A, unsigned long long __B)
{
  return (__m128d) __builtin_ia32_cvtusi2sd64 ((__v2df) __A, __B,
            0x04);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtu32_ss (__m128 __A, unsigned __B)
{
  return (__m128) __builtin_ia32_cvtusi2ss32 ((__v4sf) __A, __B,
           0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi32_ps (__m512i __A)
{
  return (__m512) __builtin_ia32_cvtdq2ps512_mask ((__v16si) __A,
         (__v16sf)
         _mm512_undefined_ps (),
         (__mmask16) -1,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi32_ps (__m512 __W, __mmask16 __U, __m512i __A)
{
  return (__m512) __builtin_ia32_cvtdq2ps512_mask ((__v16si) __A,
         (__v16sf) __W,
         (__mmask16) __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi32_ps (__mmask16 __U, __m512i __A)
{
  return (__m512) __builtin_ia32_cvtdq2ps512_mask ((__v16si) __A,
         (__v16sf)
         _mm512_setzero_ps (),
         (__mmask16) __U,
         0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu32_ps (__m512i __A)
{
  return (__m512) __builtin_ia32_cvtudq2ps512_mask ((__v16si) __A,
          (__v16sf)
          _mm512_undefined_ps (),
          (__mmask16) -1,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu32_ps (__m512 __W, __mmask16 __U, __m512i __A)
{
  return (__m512) __builtin_ia32_cvtudq2ps512_mask ((__v16si) __A,
          (__v16sf) __W,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu32_ps (__mmask16 __U, __m512i __A)
{
  return (__m512) __builtin_ia32_cvtudq2ps512_mask ((__v16si) __A,
          (__v16sf)
          _mm512_setzero_ps (),
          (__mmask16) __U,
          0x04);
}
# 13354 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_u64 (__m128 __A)
{
  return (unsigned long long) __builtin_ia32_vcvtss2usi64 ((__v4sf)
          __A,
          0x04);
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_u64 (__m128 __A)
{
  return (unsigned long long) __builtin_ia32_vcvttss2usi64 ((__v4sf)
           __A,
           0x04);
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_i64 (__m128 __A)
{
  return (long long) __builtin_ia32_vcvttss2si64 ((__v4sf) __A,
        0x04);
}
extern __inline unsigned
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtss_u32 (__m128 __A)
{
  return (unsigned) __builtin_ia32_vcvtss2usi32 ((__v4sf) __A,
       0x04);
}
extern __inline unsigned
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_u32 (__m128 __A)
{
  return (unsigned) __builtin_ia32_vcvttss2usi32 ((__v4sf) __A,
        0x04);
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttss_i32 (__m128 __A)
{
  return (int) __builtin_ia32_vcvttss2si32 ((__v4sf) __A,
         0x04);
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_u64 (__m128d __A)
{
  return (unsigned long long) __builtin_ia32_vcvtsd2usi64 ((__v2df)
          __A,
          0x04);
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_u64 (__m128d __A)
{
  return (unsigned long long) __builtin_ia32_vcvttsd2usi64 ((__v2df)
           __A,
           0x04);
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_i64 (__m128d __A)
{
  return (long long) __builtin_ia32_vcvttsd2si64 ((__v2df) __A,
        0x04);
}
extern __inline unsigned
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsd_u32 (__m128d __A)
{
  return (unsigned) __builtin_ia32_vcvtsd2usi32 ((__v2df) __A,
       0x04);
}
extern __inline unsigned
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_u32 (__m128d __A)
{
  return (unsigned) __builtin_ia32_vcvttsd2usi32 ((__v2df) __A,
        0x04);
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttsd_i32 (__m128d __A)
{
  return (int) __builtin_ia32_vcvttsd2si32 ((__v2df) __A,
         0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtps_pd (__m256 __A)
{
  return (__m512d) __builtin_ia32_cvtps2pd512_mask ((__v8sf) __A,
          (__v8df)
          _mm512_undefined_pd (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtps_pd (__m512d __W, __mmask8 __U, __m256 __A)
{
  return (__m512d) __builtin_ia32_cvtps2pd512_mask ((__v8sf) __A,
          (__v8df) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtps_pd (__mmask8 __U, __m256 __A)
{
  return (__m512d) __builtin_ia32_cvtps2pd512_mask ((__v8sf) __A,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtph_ps (__m256i __A)
{
  return (__m512) __builtin_ia32_vcvtph2ps512_mask ((__v16hi) __A,
          (__v16sf)
          _mm512_undefined_ps (),
          (__mmask16) -1,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtph_ps (__m512 __W, __mmask16 __U, __m256i __A)
{
  return (__m512) __builtin_ia32_vcvtph2ps512_mask ((__v16hi) __A,
          (__v16sf) __W,
          (__mmask16) __U,
          0x04);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtph_ps (__mmask16 __U, __m256i __A)
{
  return (__m512) __builtin_ia32_vcvtph2ps512_mask ((__v16hi) __A,
          (__v16sf)
          _mm512_setzero_ps (),
          (__mmask16) __U,
          0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtpd_ps (__m512d __A)
{
  return (__m256) __builtin_ia32_cvtpd2ps512_mask ((__v8df) __A,
         (__v8sf)
         _mm256_undefined_ps (),
         (__mmask8) -1,
         0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtpd_ps (__m256 __W, __mmask8 __U, __m512d __A)
{
  return (__m256) __builtin_ia32_cvtpd2ps512_mask ((__v8df) __A,
         (__v8sf) __W,
         (__mmask8) __U,
         0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtpd_ps (__mmask8 __U, __m512d __A)
{
  return (__m256) __builtin_ia32_cvtpd2ps512_mask ((__v8df) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U,
         0x04);
}
# 14529 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kmov (__mmask16 __A)
{
  return __builtin_ia32_kmovw (__A);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd_ps (__m512d __A)
{
  return (__m512) (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd_si512 (__m512d __A)
{
  return (__m512i) (__A);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps_pd (__m512 __A)
{
  return (__m512d) (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps_si512 (__m512 __A)
{
  return (__m512i) (__A);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi512_ps (__m512i __A)
{
  return (__m512) (__A);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi512_pd (__m512i __A)
{
  return (__m512d) (__A);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd512_pd128 (__m512d __A)
{
  return (__m128d)((__m128) __builtin_ia32_extractf32x4_mask ((__v16sf)(__m512) ((__m512)__A), (int) (0), (__v4sf)(__m128)_mm_undefined_ps(), (__mmask8)-1));
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps512_ps128 (__m512 __A)
{
  return ((__m128) __builtin_ia32_extractf32x4_mask ((__v16sf)(__m512) (__A), (int) (0), (__v4sf)(__m128)_mm_undefined_ps(), (__mmask8)-1));
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi512_si128 (__m512i __A)
{
  return (__m128i)((__m128i) __builtin_ia32_extracti32x4_mask ((__v16si)(__m512i) ((__m512i)__A), (int) (0), (__v4si)(__m128i)_mm_undefined_si128 (), (__mmask8)-1));
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd512_pd256 (__m512d __A)
{
  return ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1));
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps512_ps256 (__m512 __A)
{
  return (__m256)((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d)__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1));
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi512_si256 (__m512i __A)
{
  return (__m256i)((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d)__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1));
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd128_pd512 (__m128d __A)
{
  return (__m512d) __builtin_ia32_pd512_pd((__m128d)__A);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps128_ps512 (__m128 __A)
{
  return (__m512) __builtin_ia32_ps512_ps((__m128)__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi128_si512 (__m128i __A)
{
  return (__m512i) __builtin_ia32_si512_si((__v4si)__A);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castpd256_pd512 (__m256d __A)
{
  return __builtin_ia32_pd512_256pd (__A);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castps256_ps512 (__m256 __A)
{
  return __builtin_ia32_ps512_256ps (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_castsi256_si512 (__m256i __A)
{
  return (__m512i)__builtin_ia32_si512_256si ((__v8si)__A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epu32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __A,
           (__v16si) __B, 0,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epu32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __A,
           (__v16si) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epu64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __A,
          (__v8di) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epu64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __A,
          (__v8di) __B, 0,
          (__mmask8) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epu32_mask (__m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __A,
           (__v16si) __B, 6,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epu32_mask (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__mmask16) __builtin_ia32_ucmpd512_mask ((__v16si) __A,
           (__v16si) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epu64_mask (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __A,
          (__v8di) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epu64_mask (__m512i __A, __m512i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq512_mask ((__v8di) __A,
          (__v8di) __B, 6,
          (__mmask8) -1);
}
# 14742 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_add_epi32 (__m512i __A)
{
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 + __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 + __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 + __T7; return __T8[0] + __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_mul_epi32 (__m512i __A)
{
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 * __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 * __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 * __T7; return __T8[0] * __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_and_epi32 (__m512i __A)
{
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 & __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 & __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 & __T7; return __T8[0] & __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_or_epi32 (__m512i __A)
{
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 | __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 | __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 | __T7; return __T8[0] | __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_add_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi32 (__U, __A);
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 + __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 + __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 + __T7; return __T8[0] + __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_mul_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi32 (_mm512_set1_epi32 (1), __U, __A);
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 * __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 * __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 * __T7; return __T8[0] * __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_and_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi32 (_mm512_set1_epi32 (~0), __U, __A);
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 & __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 & __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 & __T7; return __T8[0] & __T8[1];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_or_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi32 (__U, __A);
  __v8si __T1 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v8si __T2 = (__v8si) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 | __T2); __v4si __T4 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v4si __T5 = (__v4si) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v4si __T6 = __T4 | __T5; __v4si __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __v4si __T8 = __T6 | __T7; return __T8[0] | __T8[1];
}
# 14818 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_epi32 (__m512i __A)
{
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_min_epi32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_min_epi32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_min_epi32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_min_epi32 (__T8, __T9); return __T10[0];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_epi32 (__m512i __A)
{
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_max_epi32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_max_epi32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_max_epi32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_max_epi32 (__T8, __T9); return __T10[0];
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_epu32 (__m512i __A)
{
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_min_epu32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_min_epu32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_min_epu32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_min_epu32 (__T8, __T9); return __T10[0];
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_epu32 (__m512i __A)
{
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_max_epu32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_max_epu32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_max_epu32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_max_epu32 (__T8, __T9); return __T10[0];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi32 (_mm512_set1_epi32 (0x7fffffff), __U, __A);
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_min_epi32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_min_epi32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_min_epi32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_min_epi32 (__T8, __T9); return __T10[0];
}
extern __inline int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_epi32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi32 (_mm512_set1_epi32 (-0x7fffffff - 1), __U, __A);
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_max_epi32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_max_epi32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_max_epi32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_max_epi32 (__T8, __T9); return __T10[0];
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_epu32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi32 (_mm512_set1_epi32 (~0), __U, __A);
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_min_epu32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_min_epu32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_min_epu32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_min_epu32 (__T8, __T9); return __T10[0];
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_epu32 (__mmask16 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi32 (__U, __A);
  __m256i __T1 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T2 = (__m256i) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = _mm256_max_epu32 (__T1, __T2); __m128i __T4 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __m128i __T5 = (__m128i) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __m128i __T6 = _mm_max_epu32 (__T4, __T5); __m128i __T7 = (__m128i) __builtin_shuffle ((__v4si) __T6, (__v4si) { 2, 3, 0, 1 }); __m128i __T8 = _mm_max_epu32 (__T6, __T7); __m128i __T9 = (__m128i) __builtin_shuffle ((__v4si) __T8, (__v4si) { 1, 0, 1, 0 }); __v4si __T10 = (__v4si) _mm_max_epu32 (__T8, __T9); return __T10[0];
}
# 14890 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_add_ps (__m512 __A)
{
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = __T1 + __T2; __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = __T4 + __T5; __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = __T6 + __T7; return __T8[0] + __T8[1];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_mul_ps (__m512 __A)
{
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = __T1 * __T2; __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = __T4 * __T5; __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = __T6 * __T7; return __T8[0] * __T8[1];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_add_ps (__mmask16 __U, __m512 __A)
{
  __A = _mm512_maskz_mov_ps (__U, __A);
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = __T1 + __T2; __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = __T4 + __T5; __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = __T6 + __T7; return __T8[0] + __T8[1];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_mul_ps (__mmask16 __U, __m512 __A)
{
  __A = _mm512_mask_mov_ps (_mm512_set1_ps (1.0f), __U, __A);
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = __T1 * __T2; __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = __T4 * __T5; __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = __T6 * __T7; return __T8[0] * __T8[1];
}
# 14934 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_ps (__m512 __A)
{
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = _mm256_min_ps (__T1, __T2); __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = _mm_min_ps (__T4, __T5); __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = _mm_min_ps (__T6, __T7); __m128 __T9 = __builtin_shuffle (__T8, (__v4si) { 1, 0, 1, 0 }); __m128 __T10 = _mm_min_ps (__T8, __T9); return __T10[0];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_ps (__m512 __A)
{
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = _mm256_max_ps (__T1, __T2); __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = _mm_max_ps (__T4, __T5); __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = _mm_max_ps (__T6, __T7); __m128 __T9 = __builtin_shuffle (__T8, (__v4si) { 1, 0, 1, 0 }); __m128 __T10 = _mm_max_ps (__T8, __T9); return __T10[0];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_ps (__mmask16 __U, __m512 __A)
{
  __A = _mm512_mask_mov_ps (_mm512_set1_ps (__builtin_inff ()), __U, __A);
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = _mm256_min_ps (__T1, __T2); __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = _mm_min_ps (__T4, __T5); __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = _mm_min_ps (__T6, __T7); __m128 __T9 = __builtin_shuffle (__T8, (__v4si) { 1, 0, 1, 0 }); __m128 __T10 = _mm_min_ps (__T8, __T9); return __T10[0];
}
extern __inline float
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_ps (__mmask16 __U, __m512 __A)
{
  __A = _mm512_mask_mov_ps (_mm512_set1_ps (-__builtin_inff ()), __U, __A);
  __m256 __T1 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T2 = (__m256) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) ((__m512d) __A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256 __T3 = _mm256_max_ps (__T1, __T2); __m128 __T4 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(1))); __m128 __T5 = ((__m128) __builtin_ia32_vextractf128_ps256 ((__v8sf)(__m256)(__T3), (int)(0))); __m128 __T6 = _mm_max_ps (__T4, __T5); __m128 __T7 = __builtin_shuffle (__T6, (__v4si) { 2, 3, 0, 1 }); __m128 __T8 = _mm_max_ps (__T6, __T7); __m128 __T9 = __builtin_shuffle (__T8, (__v4si) { 1, 0, 1, 0 }); __m128 __T10 = _mm_max_ps (__T8, __T9); return __T10[0];
}
# 14974 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_add_epi64 (__m512i __A)
{
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 + __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 + __T5; return __T6[0] + __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_mul_epi64 (__m512i __A)
{
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 * __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 * __T5; return __T6[0] * __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_and_epi64 (__m512i __A)
{
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 & __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 & __T5; return __T6[0] & __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_or_epi64 (__m512i __A)
{
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 | __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 | __T5; return __T6[0] | __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_add_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi64 (__U, __A);
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 + __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 + __T5; return __T6[0] + __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_mul_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi64 (_mm512_set1_epi64 (1LL), __U, __A);
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 * __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 * __T5; return __T6[0] * __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_and_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi64 (_mm512_set1_epi64 (~0LL), __U, __A);
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 & __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 & __T5; return __T6[0] & __T6[1];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_or_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi64 (__U, __A);
  __v4di __T1 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (1), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __v4di __T2 = (__v4di) ((__m256i) __builtin_ia32_extracti64x4_mask ((__v8di)(__m512i) (__A), (int) (0), (__v4di)(__m256i)_mm256_undefined_si256 (), (__mmask8)-1)); __m256i __T3 = (__m256i) (__T1 | __T2); __v2di __T4 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(1))); __v2di __T5 = (__v2di) ((__m128i) __builtin_ia32_extract128i256 ((__v4di)(__m256i)(__T3), (int)(0))); __v2di __T6 = __T4 | __T5; return __T6[0] | __T6[1];
}
# 15048 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_epi64 (__m512i __A)
{
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_min_epi64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_min_epi64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_min_epi64 (__T4, __T5); return __T6[0];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_epi64 (__m512i __A)
{
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_max_epi64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_max_epi64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_max_epi64 (__T4, __T5); return __T6[0];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi64 (_mm512_set1_epi64 (0x7fffffffffffffffLL),
          __U, __A);
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_min_epi64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_min_epi64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_min_epi64 (__T4, __T5); return __T6[0];
}
extern __inline long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_epi64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi64 (_mm512_set1_epi64 (-0x7fffffffffffffffLL - 1),
          __U, __A);
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_max_epi64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_max_epi64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_max_epi64 (__T4, __T5); return __T6[0];
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_epu64 (__m512i __A)
{
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_min_epu64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_min_epu64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_min_epu64 (__T4, __T5); return __T6[0];
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_epu64 (__m512i __A)
{
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_max_epu64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_max_epu64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_max_epu64 (__T4, __T5); return __T6[0];
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_epu64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_mask_mov_epi64 (_mm512_set1_epi64 (~0LL), __U, __A);
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_min_epu64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_min_epu64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_min_epu64 (__T4, __T5); return __T6[0];
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_epu64 (__mmask8 __U, __m512i __A)
{
  __A = _mm512_maskz_mov_epi64 (__U, __A);
  __m512i __T1 = ((__m512i) __builtin_ia32_shuf_i64x2_mask ((__v8di)(__m512i)(__A), (__v8di)(__m512i)(__A), (int)(0x4e), (__v8di)(__m512i)_mm512_undefined_epi32 (), (__mmask8)-1)); __m512i __T2 = _mm512_max_epu64 (__A, __T1); __m512i __T3 = (__m512i) __builtin_shuffle ((__v8di) __T2, (__v8di) { 2, 3, 0, 1, 6, 7, 4, 5 }); __m512i __T4 = _mm512_max_epu64 (__T2, __T3); __m512i __T5 = (__m512i) __builtin_shuffle ((__v8di) __T4, (__v8di) { 1, 0, 3, 2, 5, 4, 7, 6 }); __v8di __T6 = (__v8di) _mm512_max_epu64 (__T4, __T5); return __T6[0];
}
# 15120 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_add_pd (__m512d __A)
{
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = __T1 + __T2; __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = __T4 + __T5; return __T6[0] + __T6[1];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_mul_pd (__m512d __A)
{
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = __T1 * __T2; __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = __T4 * __T5; return __T6[0] * __T6[1];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_add_pd (__mmask8 __U, __m512d __A)
{
  __A = _mm512_maskz_mov_pd (__U, __A);
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = __T1 + __T2; __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = __T4 + __T5; return __T6[0] + __T6[1];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_mul_pd (__mmask8 __U, __m512d __A)
{
  __A = _mm512_mask_mov_pd (_mm512_set1_pd (1.0), __U, __A);
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = __T1 * __T2; __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = __T4 * __T5; return __T6[0] * __T6[1];
}
# 15162 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512fintrin.h" 3 4
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_min_pd (__m512d __A)
{
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = _mm256_min_pd (__T1, __T2); __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = _mm_min_pd (__T4, __T5); __m128d __T7 = (__m128d) __builtin_shuffle (__T6, (__v2di) { 1, 0 }); __m128d __T8 = _mm_min_pd (__T6, __T7); return __T8[0];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_reduce_max_pd (__m512d __A)
{
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = _mm256_max_pd (__T1, __T2); __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = _mm_max_pd (__T4, __T5); __m128d __T7 = (__m128d) __builtin_shuffle (__T6, (__v2di) { 1, 0 }); __m128d __T8 = _mm_max_pd (__T6, __T7); return __T8[0];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_min_pd (__mmask8 __U, __m512d __A)
{
  __A = _mm512_mask_mov_pd (_mm512_set1_pd (__builtin_inf ()), __U, __A);
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = _mm256_min_pd (__T1, __T2); __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = _mm_min_pd (__T4, __T5); __m128d __T7 = (__m128d) __builtin_shuffle (__T6, (__v2di) { 1, 0 }); __m128d __T8 = _mm_min_pd (__T6, __T7); return __T8[0];
}
extern __inline double
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_reduce_max_pd (__mmask8 __U, __m512d __A)
{
  __A = _mm512_mask_mov_pd (_mm512_set1_pd (-__builtin_inf ()), __U, __A);
  __m256d __T1 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (1), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T2 = (__m256d) ((__m256d) __builtin_ia32_extractf64x4_mask ((__v8df)(__m512d) (__A), (int) (0), (__v4df)(__m256d)_mm256_undefined_pd(), (__mmask8)-1)); __m256d __T3 = _mm256_max_pd (__T1, __T2); __m128d __T4 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(1))); __m128d __T5 = ((__m128d) __builtin_ia32_vextractf128_pd256 ((__v4df)(__m256d)(__T3), (int)(0))); __m128d __T6 = _mm_max_pd (__T4, __T5); __m128d __T7 = (__m128d) __builtin_shuffle (__T6, (__v2di) { 1, 0 }); __m128d __T8 = _mm_max_pd (__T6, __T7); return __T8[0];
}
#pragma GCC pop_options
# 46 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512erintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512erintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512er")
typedef double __v8df __attribute__ ((__vector_size__ (64)));
typedef float __v16sf __attribute__ ((__vector_size__ (64)));
typedef float __m512 __attribute__ ((__vector_size__ (64), __may_alias__));
typedef double __m512d __attribute__ ((__vector_size__ (64), __may_alias__));
typedef unsigned char __mmask8;
typedef unsigned short __mmask16;
# 391 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512erintrin.h" 3 4
#pragma GCC pop_options
# 48 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512pfintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512pfintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512pf")
typedef long long __v8di __attribute__ ((__vector_size__ (64)));
typedef int __v16si __attribute__ ((__vector_size__ (64)));
typedef long long __m512i __attribute__ ((__vector_size__ (64), __may_alias__));
typedef unsigned char __mmask8;
typedef unsigned short __mmask16;
# 261 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512pfintrin.h" 3 4
#pragma GCC pop_options
# 50 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512cdintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512cdintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512cd")
typedef long long __v8di __attribute__ ((__vector_size__ (64)));
typedef int __v16si __attribute__ ((__vector_size__ (64)));
typedef long long __m512i __attribute__ ((__vector_size__ (64), __may_alias__));
typedef double __m512d __attribute__ ((__vector_size__ (64), __may_alias__));
typedef unsigned char __mmask8;
typedef unsigned short __mmask16;
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_conflict_epi32 (__m512i __A)
{
  return (__m512i)
  __builtin_ia32_vpconflictsi_512_mask ((__v16si) __A,
            (__v16si) _mm512_setzero_si512 (),
            (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_conflict_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpconflictsi_512_mask ((__v16si) __A,
        (__v16si) __W,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_conflict_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i)
  __builtin_ia32_vpconflictsi_512_mask ((__v16si) __A,
            (__v16si) _mm512_setzero_si512 (),
            (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_conflict_epi64 (__m512i __A)
{
  return (__m512i)
  __builtin_ia32_vpconflictdi_512_mask ((__v8di) __A,
            (__v8di) _mm512_setzero_si512 (),
            (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_conflict_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpconflictdi_512_mask ((__v8di) __A,
        (__v8di) __W,
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_conflict_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i)
  __builtin_ia32_vpconflictdi_512_mask ((__v8di) __A,
            (__v8di) _mm512_setzero_si512 (),
            (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_lzcnt_epi64 (__m512i __A)
{
  return (__m512i)
  __builtin_ia32_vplzcntq_512_mask ((__v8di) __A,
        (__v8di) _mm512_setzero_si512 (),
        (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_lzcnt_epi64 (__m512i __W, __mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vplzcntq_512_mask ((__v8di) __A,
           (__v8di) __W,
           (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_lzcnt_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i)
  __builtin_ia32_vplzcntq_512_mask ((__v8di) __A,
        (__v8di) _mm512_setzero_si512 (),
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_lzcnt_epi32 (__m512i __A)
{
  return (__m512i)
  __builtin_ia32_vplzcntd_512_mask ((__v16si) __A,
        (__v16si) _mm512_setzero_si512 (),
        (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_lzcnt_epi32 (__m512i __W, __mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vplzcntd_512_mask ((__v16si) __A,
           (__v16si) __W,
           (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_lzcnt_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i)
  __builtin_ia32_vplzcntd_512_mask ((__v16si) __A,
        (__v16si) _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastmb_epi64 (__mmask8 __A)
{
  return (__m512i) __builtin_ia32_broadcastmb512 (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastmw_epi32 (__mmask16 __A)
{
  return (__m512i) __builtin_ia32_broadcastmw512 (__A);
}
#pragma GCC pop_options
# 52 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vl")
typedef unsigned int __mmask32;
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_movapd256_mask ((__v4df) __A,
        (__v4df) __W,
        (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_movapd256_mask ((__v4df) __A,
        (__v4df)
        _mm256_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_movapd128_mask ((__v2df) __A,
        (__v2df) __W,
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_movapd128_mask ((__v2df) __A,
        (__v2df)
        _mm_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_load_pd (__m256d __W, __mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_loadapd256_mask ((__v4df *) __P,
         (__v4df) __W,
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_load_pd (__mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_loadapd256_mask ((__v4df *) __P,
         (__v4df)
         _mm256_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_load_pd (__m128d __W, __mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_loadapd128_mask ((__v2df *) __P,
         (__v2df) __W,
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_load_pd (__mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_loadapd128_mask ((__v2df *) __P,
         (__v2df)
         _mm_setzero_pd (),
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_store_pd (void *__P, __mmask8 __U, __m256d __A)
{
  __builtin_ia32_storeapd256_mask ((__v4df *) __P,
       (__v4df) __A,
       (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_store_pd (void *__P, __mmask8 __U, __m128d __A)
{
  __builtin_ia32_storeapd128_mask ((__v2df *) __P,
       (__v2df) __A,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movaps256_mask ((__v8sf) __A,
       (__v8sf) __W,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movaps256_mask ((__v8sf) __A,
       (__v8sf)
       _mm256_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movaps128_mask ((__v4sf) __A,
       (__v4sf) __W,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movaps128_mask ((__v4sf) __A,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_load_ps (__m256 __W, __mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_loadaps256_mask ((__v8sf *) __P,
        (__v8sf) __W,
        (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_load_ps (__mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_loadaps256_mask ((__v8sf *) __P,
        (__v8sf)
        _mm256_setzero_ps (),
        (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_load_ps (__m128 __W, __mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_loadaps128_mask ((__v4sf *) __P,
        (__v4sf) __W,
        (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_load_ps (__mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_loadaps128_mask ((__v4sf *) __P,
        (__v4sf)
        _mm_setzero_ps (),
        (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_store_ps (void *__P, __mmask8 __U, __m256 __A)
{
  __builtin_ia32_storeaps256_mask ((__v8sf *) __P,
       (__v8sf) __A,
       (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_store_ps (void *__P, __mmask8 __U, __m128 __A)
{
  __builtin_ia32_storeaps128_mask ((__v4sf *) __P,
       (__v4sf) __A,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdqa64_256_mask ((__v4di) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdqa64_256_mask ((__v4di) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdqa64_128_mask ((__v2di) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdqa64_128_mask ((__v2di) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_load_epi64 (__m256i __W, __mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_movdqa64load256_mask ((__v4di *) __P,
       (__v4di) __W,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_load_epi64 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_movdqa64load256_mask ((__v4di *) __P,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_load_epi64 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_movdqa64load128_mask ((__v2di *) __P,
       (__v2di) __W,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_load_epi64 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_movdqa64load128_mask ((__v2di *) __P,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8)
       __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_store_epi64 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_movdqa64store256_mask ((__v4di *) __P,
     (__v4di) __A,
     (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_store_epi64 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_movdqa64store128_mask ((__v2di *) __P,
     (__v2di) __A,
     (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdqa32_256_mask ((__v8si) __A,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdqa32_256_mask ((__v8si) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdqa32_128_mask ((__v4si) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdqa32_128_mask ((__v4si) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_load_epi32 (__m256i __W, __mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_movdqa32load256_mask ((__v8si *) __P,
       (__v8si) __W,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_load_epi32 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_movdqa32load256_mask ((__v8si *) __P,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_load_epi32 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_movdqa32load128_mask ((__v4si *) __P,
       (__v4si) __W,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_load_epi32 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_movdqa32load128_mask ((__v4si *) __P,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8)
       __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_store_epi32 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_movdqa32store256_mask ((__v8si *) __P,
     (__v8si) __A,
     (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_store_epi32 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_movdqa32store128_mask ((__v4si *) __P,
     (__v4si) __A,
     (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_addpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_addpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_addpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_addpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_addps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_addps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_addps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_addps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_subpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_subpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_subpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_subpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_subps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_subps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_subps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_subps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_store_epi64 (void *__P, __m256i __A)
{
  *(__m256i *) __P = __A;
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_store_epi64 (void *__P, __m128i __A)
{
  *(__m128i *) __P = __A;
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_pd (__m256d __W, __mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_loadupd256_mask ((const double *) __P,
         (__v4df) __W,
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_pd (__mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_loadupd256_mask ((const double *) __P,
         (__v4df)
         _mm256_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_pd (__m128d __W, __mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_loadupd128_mask ((const double *) __P,
         (__v2df) __W,
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_pd (__mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_loadupd128_mask ((const double *) __P,
         (__v2df)
         _mm_setzero_pd (),
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_pd (void *__P, __mmask8 __U, __m256d __A)
{
  __builtin_ia32_storeupd256_mask ((double *) __P,
       (__v4df) __A,
       (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_pd (void *__P, __mmask8 __U, __m128d __A)
{
  __builtin_ia32_storeupd128_mask ((double *) __P,
       (__v2df) __A,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_ps (__m256 __W, __mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_loadups256_mask ((const float *) __P,
        (__v8sf) __W,
        (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_ps (__mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_loadups256_mask ((const float *) __P,
        (__v8sf)
        _mm256_setzero_ps (),
        (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_ps (__m128 __W, __mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_loadups128_mask ((const float *) __P,
        (__v4sf) __W,
        (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_ps (__mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_loadups128_mask ((const float *) __P,
        (__v4sf)
        _mm_setzero_ps (),
        (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_ps (void *__P, __mmask8 __U, __m256 __A)
{
  __builtin_ia32_storeups256_mask ((float *) __P,
       (__v8sf) __A,
       (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_ps (void *__P, __mmask8 __U, __m128 __A)
{
  __builtin_ia32_storeups128_mask ((float *) __P,
       (__v4sf) __A,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_epi64 (__m256i __W, __mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddqudi256_mask ((const long long *) __P,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddqudi256_mask ((const long long *) __P,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_epi64 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddqudi128_mask ((const long long *) __P,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddqudi128_mask ((const long long *) __P,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_epi64 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_storedqudi256_mask ((long long *) __P,
         (__v4di) __A,
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_epi64 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_storedqudi128_mask ((long long *) __P,
         (__v2di) __A,
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_epi32 (__m256i __W, __mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddqusi256_mask ((const int *) __P,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_epi32 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddqusi256_mask ((const int *) __P,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_epi32 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddqusi128_mask ((const int *) __P,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_epi32 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddqusi128_mask ((const int *) __P,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_epi32 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_storedqusi256_mask ((int *) __P,
         (__v8si) __A,
         (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_epi32 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_storedqusi128_mask ((int *) __P,
         (__v4si) __A,
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_abs_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsd256_mask ((__v8si) __A,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_abs_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsd256_mask ((__v8si) __A,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_abs_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsd128_mask ((__v4si) __A,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_abs_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsd128_mask ((__v4si) __A,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_abs_epi64 (__m256i __A)
{
  return (__m256i) __builtin_ia32_pabsq256_mask ((__v4di) __A,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_abs_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsq256_mask ((__v4di) __A,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_abs_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsq256_mask ((__v4di) __A,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_abs_epi64 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pabsq128_mask ((__v2di) __A,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_abs_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsq128_mask ((__v2di) __A,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_abs_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsq128_mask ((__v2di) __A,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtpd_epu32 (__m256d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq256_mask ((__v4df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtpd_epu32 (__m128i __W, __mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq256_mask ((__v4df) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtpd_epu32 (__mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq256_mask ((__v4df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_epu32 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq128_mask ((__v2df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtpd_epu32 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq128_mask ((__v2df) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtpd_epu32 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2udq128_mask ((__v2df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttps_epi32 (__m256i __W, __mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvttps2dq256_mask ((__v8sf) __A,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttps_epi32 (__mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvttps2dq256_mask ((__v8sf) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttps_epi32 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2dq128_mask ((__v4sf) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttps_epi32 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2dq128_mask ((__v4sf) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttps_epu32 (__m256 __A)
{
  return (__m256i) __builtin_ia32_cvttps2udq256_mask ((__v8sf) __A,
            (__v8si)
            _mm256_setzero_si256 (),
            (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttps_epu32 (__m256i __W, __mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvttps2udq256_mask ((__v8sf) __A,
            (__v8si) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttps_epu32 (__mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvttps2udq256_mask ((__v8sf) __A,
            (__v8si)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttps_epu32 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2udq128_mask ((__v4sf) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttps_epu32 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2udq128_mask ((__v4sf) __A,
            (__v4si) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttps_epu32 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2udq128_mask ((__v4sf) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttpd_epi32 (__m128i __W, __mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2dq256_mask ((__v4df) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttpd_epi32 (__mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2dq256_mask ((__v4df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttpd_epi32 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2dq128_mask ((__v2df) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttpd_epi32 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2dq128_mask ((__v2df) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttpd_epu32 (__m256d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq256_mask ((__v4df) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttpd_epu32 (__m128i __W, __mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq256_mask ((__v4df) __A,
            (__v4si) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttpd_epu32 (__mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq256_mask ((__v4df) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttpd_epu32 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq128_mask ((__v2df) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttpd_epu32 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq128_mask ((__v2df) __A,
            (__v4si) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttpd_epu32 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2udq128_mask ((__v2df) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtpd_epi32 (__m128i __W, __mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2dq256_mask ((__v4df) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtpd_epi32 (__mmask8 __U, __m256d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2dq256_mask ((__v4df) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtpd_epi32 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2dq128_mask ((__v2df) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtpd_epi32 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2dq128_mask ((__v2df) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_pd (__m256d __W, __mmask8 __U, __m128i __A)
{
  return (__m256d) __builtin_ia32_cvtdq2pd256_mask ((__v4si) __A,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi32_pd (__mmask8 __U, __m128i __A)
{
  return (__m256d) __builtin_ia32_cvtdq2pd256_mask ((__v4si) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_pd (__m128d __W, __mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtdq2pd128_mask ((__v4si) __A,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi32_pd (__mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtdq2pd128_mask ((__v4si) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu32_pd (__m128i __A)
{
  return (__m256d) __builtin_ia32_cvtudq2pd256_mask ((__v4si) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu32_pd (__m256d __W, __mmask8 __U, __m128i __A)
{
  return (__m256d) __builtin_ia32_cvtudq2pd256_mask ((__v4si) __A,
           (__v4df) __W,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu32_pd (__mmask8 __U, __m128i __A)
{
  return (__m256d) __builtin_ia32_cvtudq2pd256_mask ((__v4si) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu32_pd (__m128i __A)
{
  return (__m128d) __builtin_ia32_cvtudq2pd128_mask ((__v4si) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu32_pd (__m128d __W, __mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtudq2pd128_mask ((__v4si) __A,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu32_pd (__mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtudq2pd128_mask ((__v4si) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_ps (__m256 __W, __mmask8 __U, __m256i __A)
{
  return (__m256) __builtin_ia32_cvtdq2ps256_mask ((__v8si) __A,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi32_ps (__mmask8 __U, __m256i __A)
{
  return (__m256) __builtin_ia32_cvtdq2ps256_mask ((__v8si) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_ps (__m128 __W, __mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtdq2ps128_mask ((__v4si) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi32_ps (__mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtdq2ps128_mask ((__v4si) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu32_ps (__m256i __A)
{
  return (__m256) __builtin_ia32_cvtudq2ps256_mask ((__v8si) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu32_ps (__m256 __W, __mmask8 __U, __m256i __A)
{
  return (__m256) __builtin_ia32_cvtudq2ps256_mask ((__v8si) __A,
          (__v8sf) __W,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu32_ps (__mmask8 __U, __m256i __A)
{
  return (__m256) __builtin_ia32_cvtudq2ps256_mask ((__v8si) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu32_ps (__m128i __A)
{
  return (__m128) __builtin_ia32_cvtudq2ps128_mask ((__v4si) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu32_ps (__m128 __W, __mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtudq2ps128_mask ((__v4si) __A,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu32_ps (__mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtudq2ps128_mask ((__v4si) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtps_pd (__m256d __W, __mmask8 __U, __m128 __A)
{
  return (__m256d) __builtin_ia32_cvtps2pd256_mask ((__v4sf) __A,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtps_pd (__mmask8 __U, __m128 __A)
{
  return (__m256d) __builtin_ia32_cvtps2pd256_mask ((__v4sf) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtps_pd (__m128d __W, __mmask8 __U, __m128 __A)
{
  return (__m128d) __builtin_ia32_cvtps2pd128_mask ((__v4sf) __A,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtps_pd (__mmask8 __U, __m128 __A)
{
  return (__m128d) __builtin_ia32_cvtps2pd128_mask ((__v4sf) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi32_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdb128_mask ((__v4si) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovdb128mem_mask ((__v16qi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdb128_mask ((__v4si) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi32_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdb128_mask ((__v4si) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi32_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdb256_mask ((__v8si) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdb256_mask ((__v8si) __A,
        (__v16qi) __O, __M);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovdb256mem_mask ((__v16qi *) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi32_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdb256_mask ((__v8si) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi32_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb128_mask ((__v4si) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi32_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovsdb128mem_mask ((__v16qi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi32_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb128_mask ((__v4si) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi32_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb128_mask ((__v4si) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi32_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb256_mask ((__v8si) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi32_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovsdb256mem_mask ((__v16qi *) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi32_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb256_mask ((__v8si) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi32_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdb256_mask ((__v8si) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi32_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb128_mask ((__v4si) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi32_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovusdb128mem_mask ((__v16qi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi32_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb128_mask ((__v4si) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi32_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb128_mask ((__v4si) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi32_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb256_mask ((__v8si) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi32_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovusdb256mem_mask ((__v16qi*) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi32_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb256_mask ((__v8si) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi32_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdb256_mask ((__v8si) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi32_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdw128_mask ((__v4si) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovdw128mem_mask ((__v8hi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdw128_mask ((__v4si) __A,
        (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi32_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovdw128_mask ((__v4si) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi32_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdw256_mask ((__v8si) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovdw256mem_mask ((__v8hi *) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdw256_mask ((__v8si) __A,
        (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi32_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovdw256_mask ((__v8si) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi32_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw128_mask ((__v4si) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi32_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovsdw128mem_mask ((__v8hi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi32_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw128_mask ((__v4si) __A,
         (__v8hi)__O,
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi32_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw128_mask ((__v4si) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi32_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw256_mask ((__v8si) __A,
         (__v8hi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi32_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovsdw256mem_mask ((__v8hi *) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi32_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw256_mask ((__v8si) __A,
         (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi32_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsdw256_mask ((__v8si) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi32_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw128_mask ((__v4si) __A,
          (__v8hi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi32_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovusdw128mem_mask ((__v8hi *) __P, (__v4si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi32_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw128_mask ((__v4si) __A,
          (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi32_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw128_mask ((__v4si) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi32_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw256_mask ((__v8si) __A,
          (__v8hi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi32_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovusdw256mem_mask ((__v8hi *) __P, (__v8si) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi32_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw256_mask ((__v8si) __A,
          (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi32_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusdw256_mask ((__v8si) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi64_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqb128_mask ((__v2di) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovqb128mem_mask ((__v16qi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqb128_mask ((__v2di) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi64_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqb128_mask ((__v2di) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi64_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqb256_mask ((__v4di) __A,
        (__v16qi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovqb256mem_mask ((__v16qi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqb256_mask ((__v4di) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi64_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqb256_mask ((__v4di) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi64_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb128_mask ((__v2di) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovsqb128mem_mask ((__v16qi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb128_mask ((__v2di) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi64_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb128_mask ((__v2di) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi64_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb256_mask ((__v4di) __A,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovsqb256mem_mask ((__v16qi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb256_mask ((__v4di) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi64_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqb256_mask ((__v4di) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi64_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb128_mask ((__v2di) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_storeu_epi8 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovusqb128mem_mask ((__v16qi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb128_mask ((__v2di) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi64_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb128_mask ((__v2di) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi64_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb256_mask ((__v4di) __A,
          (__v16qi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_storeu_epi8 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovusqb256mem_mask ((__v16qi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_epi8 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb256_mask ((__v4di) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi64_epi8 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqb256_mask ((__v4di) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi64_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqw128_mask ((__v2di) __A,
        (__v8hi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovqw128mem_mask ((__v8hi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqw128_mask ((__v2di) __A,
        (__v8hi)__O,
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi64_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqw128_mask ((__v2di) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi64_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqw256_mask ((__v4di) __A,
        (__v8hi)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovqw256mem_mask ((__v8hi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqw256_mask ((__v4di) __A,
        (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi64_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqw256_mask ((__v4di) __A,
        (__v8hi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi64_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw128_mask ((__v2di) __A,
         (__v8hi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovsqw128mem_mask ((__v8hi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw128_mask ((__v2di) __A,
         (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi64_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw128_mask ((__v2di) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi64_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw256_mask ((__v4di) __A,
         (__v8hi)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovsqw256mem_mask ((__v8hi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw256_mask ((__v4di) __A,
         (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi64_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqw256_mask ((__v4di) __A,
         (__v8hi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi64_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw128_mask ((__v2di) __A,
          (__v8hi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_storeu_epi16 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovusqw128mem_mask ((__v8hi *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw128_mask ((__v2di) __A,
          (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi64_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw128_mask ((__v2di) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi64_epi16 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw256_mask ((__v4di) __A,
          (__v8hi)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_storeu_epi16 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovusqw256mem_mask ((__v8hi *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_epi16 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw256_mask ((__v4di) __A,
          (__v8hi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi64_epi16 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqw256_mask ((__v4di) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi64_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqd128_mask ((__v2di) __A,
        (__v4si)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_storeu_epi32 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovqd128mem_mask ((__v4si *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_epi32 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqd128_mask ((__v2di) __A,
        (__v4si) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi64_epi32 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovqd128_mask ((__v2di) __A,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi64_epi32 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqd256_mask ((__v4di) __A,
        (__v4si)
        _mm_undefined_si128 (),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_storeu_epi32 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovqd256mem_mask ((__v4si *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_epi32 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqd256_mask ((__v4di) __A,
        (__v4si) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi64_epi32 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovqd256_mask ((__v4di) __A,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi64_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd128_mask ((__v2di) __A,
         (__v4si)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_storeu_epi32 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovsqd128mem_mask ((__v4si *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi64_epi32 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd128_mask ((__v2di) __A,
         (__v4si) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi64_epi32 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd128_mask ((__v2di) __A,
         (__v4si)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi64_epi32 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd256_mask ((__v4di) __A,
         (__v4si)
         _mm_undefined_si128 (),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_storeu_epi32 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovsqd256mem_mask ((__v4si *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi64_epi32 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd256_mask ((__v4di) __A,
         (__v4si)__O,
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi64_epi32 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovsqd256_mask ((__v4di) __A,
         (__v4si)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi64_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd128_mask ((__v2di) __A,
          (__v4si)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_storeu_epi32 (void * __P, __mmask8 __M, __m128i __A)
{
  __builtin_ia32_pmovusqd128mem_mask ((__v4si *) __P, (__v2di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi64_epi32 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd128_mask ((__v2di) __A,
          (__v4si) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi64_epi32 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd128_mask ((__v2di) __A,
          (__v4si)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi64_epi32 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd256_mask ((__v4di) __A,
          (__v4si)
          _mm_undefined_si128 (),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_storeu_epi32 (void * __P, __mmask8 __M, __m256i __A)
{
  __builtin_ia32_pmovusqd256mem_mask ((__v4si *) __P, (__v4di) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi64_epi32 (__m128i __O, __mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd256_mask ((__v4di) __A,
          (__v4si) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi64_epi32 (__mmask8 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovusqd256_mask ((__v4di) __A,
          (__v4si)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastss_ps (__m256 __O, __mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastss256_mask ((__v4sf) __A,
            (__v8sf) __O,
            __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastss_ps (__mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastss256_mask ((__v4sf) __A,
            (__v8sf)
            _mm256_setzero_ps (),
            __M);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcastss_ps (__m128 __O, __mmask8 __M, __m128 __A)
{
  return (__m128) __builtin_ia32_broadcastss128_mask ((__v4sf) __A,
            (__v4sf) __O,
            __M);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcastss_ps (__mmask8 __M, __m128 __A)
{
  return (__m128) __builtin_ia32_broadcastss128_mask ((__v4sf) __A,
            (__v4sf)
            _mm_setzero_ps (),
            __M);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastsd_pd (__m256d __O, __mmask8 __M, __m128d __A)
{
  return (__m256d) __builtin_ia32_broadcastsd256_mask ((__v2df) __A,
             (__v4df) __O,
             __M);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastsd_pd (__mmask8 __M, __m128d __A)
{
  return (__m256d) __builtin_ia32_broadcastsd256_mask ((__v2df) __A,
             (__v4df)
             _mm256_setzero_pd (),
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastd_epi32 (__m256i __O, __mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastd256_mask ((__v4si) __A,
             (__v8si) __O,
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastd_epi32 (__mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastd256_mask ((__v4si) __A,
             (__v8si)
             _mm256_setzero_si256 (),
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_set1_epi32 (__m256i __O, __mmask8 __M, int __A)
{
  return (__m256i) __builtin_ia32_pbroadcastd256_gpr_mask (__A, (__v8si) __O,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_set1_epi32 (__mmask8 __M, int __A)
{
  return (__m256i) __builtin_ia32_pbroadcastd256_gpr_mask (__A,
          (__v8si)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcastd_epi32 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastd128_mask ((__v4si) __A,
             (__v4si) __O,
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcastd_epi32 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastd128_mask ((__v4si) __A,
             (__v4si)
             _mm_setzero_si128 (),
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_set1_epi32 (__m128i __O, __mmask8 __M, int __A)
{
  return (__m128i) __builtin_ia32_pbroadcastd128_gpr_mask (__A, (__v4si) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_set1_epi32 (__mmask8 __M, int __A)
{
  return (__m128i)
  __builtin_ia32_pbroadcastd128_gpr_mask (__A,
       (__v4si) _mm_setzero_si128 (),
       __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastq_epi64 (__m256i __O, __mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastq256_mask ((__v2di) __A,
             (__v4di) __O,
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastq_epi64 (__mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastq256_mask ((__v2di) __A,
             (__v4di)
             _mm256_setzero_si256 (),
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_set1_epi64 (__m256i __O, __mmask8 __M, long long __A)
{
  return (__m256i) __builtin_ia32_pbroadcastq256_gpr_mask (__A, (__v4di) __O,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_set1_epi64 (__mmask8 __M, long long __A)
{
  return (__m256i) __builtin_ia32_pbroadcastq256_gpr_mask (__A,
          (__v4di)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcastq_epi64 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastq128_mask ((__v2di) __A,
             (__v2di) __O,
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcastq_epi64 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastq128_mask ((__v2di) __A,
             (__v2di)
             _mm_setzero_si128 (),
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_set1_epi64 (__m128i __O, __mmask8 __M, long long __A)
{
  return (__m128i) __builtin_ia32_pbroadcastq128_gpr_mask (__A, (__v2di) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_set1_epi64 (__mmask8 __M, long long __A)
{
  return (__m128i)
  __builtin_ia32_pbroadcastq128_gpr_mask (__A,
       (__v2di) _mm_setzero_si128 (),
       __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_f32x4 (__m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x4_256_mask ((__v4sf) __A,
                (__v8sf)_mm256_undefined_pd (),
         (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_f32x4 (__m256 __O, __mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x4_256_mask ((__v4sf) __A,
         (__v8sf) __O,
         __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_f32x4 (__mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x4_256_mask ((__v4sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_i32x4 (__m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x4_256_mask ((__v4si)
          __A,
                 (__v8si)_mm256_undefined_si256 (),
          (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_i32x4 (__m256i __O, __mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x4_256_mask ((__v4si)
          __A,
          (__v8si)
          __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_i32x4 (__mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x4_256_mask ((__v4si)
          __A,
          (__v8si)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi8_epi32 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbd256_mask ((__v16qi) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi8_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbd256_mask ((__v16qi) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi8_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbd128_mask ((__v16qi) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi8_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbd128_mask ((__v16qi) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi8_epi64 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbq256_mask ((__v16qi) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbq256_mask ((__v16qi) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi8_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbq128_mask ((__v16qi) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbq128_mask ((__v16qi) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi16_epi32 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxwd256_mask ((__v8hi) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi16_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxwd256_mask ((__v8hi) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi16_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxwd128_mask ((__v8hi) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi16_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxwd128_mask ((__v8hi) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi16_epi64 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxwq256_mask ((__v8hi) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxwq256_mask ((__v8hi) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi16_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxwq128_mask ((__v8hi) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxwq128_mask ((__v8hi) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi32_epi64 (__m256i __W, __mmask8 __U, __m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxdq256_mask ((__v4si) __X,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi32_epi64 (__mmask8 __U, __m128i __X)
{
  return (__m256i) __builtin_ia32_pmovsxdq256_mask ((__v4si) __X,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi32_epi64 (__m128i __W, __mmask8 __U, __m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxdq128_mask ((__v4si) __X,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi32_epi64 (__mmask8 __U, __m128i __X)
{
  return (__m128i) __builtin_ia32_pmovsxdq128_mask ((__v4si) __X,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu8_epi32 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbd256_mask ((__v16qi) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu8_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbd256_mask ((__v16qi) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu8_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbd128_mask ((__v16qi) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu8_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbd128_mask ((__v16qi) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu8_epi64 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbq256_mask ((__v16qi) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbq256_mask ((__v16qi) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu8_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbq128_mask ((__v16qi) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu8_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbq128_mask ((__v16qi) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu16_epi32 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxwd256_mask ((__v8hi) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu16_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxwd256_mask ((__v8hi) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu16_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxwd128_mask ((__v8hi) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu16_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxwd128_mask ((__v8hi) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu16_epi64 (__m256i __W, __mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxwq256_mask ((__v8hi) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxwq256_mask ((__v8hi) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu16_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxwq128_mask ((__v8hi) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu16_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxwq128_mask ((__v8hi) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu32_epi64 (__m256i __W, __mmask8 __U, __m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxdq256_mask ((__v4si) __X,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu32_epi64 (__mmask8 __U, __m128i __X)
{
  return (__m256i) __builtin_ia32_pmovzxdq256_mask ((__v4si) __X,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu32_epi64 (__m128i __W, __mmask8 __U, __m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxdq128_mask ((__v4si) __X,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu32_epi64 (__mmask8 __U, __m128i __X)
{
  return (__m128i) __builtin_ia32_pmovzxdq128_mask ((__v4si) __X,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rcp14_pd (__m256d __A)
{
  return (__m256d) __builtin_ia32_rcp14pd256_mask ((__v4df) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rcp14_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_rcp14pd256_mask ((__v4df) __A,
           (__v4df) __W,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rcp14_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_rcp14pd256_mask ((__v4df) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp14_pd (__m128d __A)
{
  return (__m128d) __builtin_ia32_rcp14pd128_mask ((__v2df) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rcp14_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_rcp14pd128_mask ((__v2df) __A,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rcp14_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_rcp14pd128_mask ((__v2df) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rcp14_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_rcp14ps256_mask ((__v8sf) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rcp14_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_rcp14ps256_mask ((__v8sf) __A,
          (__v8sf) __W,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rcp14_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_rcp14ps256_mask ((__v8sf) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rcp14_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_rcp14ps128_mask ((__v4sf) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rcp14_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_rcp14ps128_mask ((__v4sf) __A,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rcp14_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_rcp14ps128_mask ((__v4sf) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rsqrt14_pd (__m256d __A)
{
  return (__m256d) __builtin_ia32_rsqrt14pd256_mask ((__v4df) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rsqrt14_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_rsqrt14pd256_mask ((__v4df) __A,
           (__v4df) __W,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rsqrt14_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_rsqrt14pd256_mask ((__v4df) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt14_pd (__m128d __A)
{
  return (__m128d) __builtin_ia32_rsqrt14pd128_mask ((__v2df) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rsqrt14_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_rsqrt14pd128_mask ((__v2df) __A,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rsqrt14_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_rsqrt14pd128_mask ((__v2df) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rsqrt14_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_rsqrt14ps256_mask ((__v8sf) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rsqrt14_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_rsqrt14ps256_mask ((__v8sf) __A,
          (__v8sf) __W,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rsqrt14_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_rsqrt14ps256_mask ((__v8sf) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rsqrt14_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_rsqrt14ps128_mask ((__v4sf) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rsqrt14_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_rsqrt14ps128_mask ((__v4sf) __A,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rsqrt14_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_rsqrt14ps128_mask ((__v4sf) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sqrt_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_sqrtpd256_mask ((__v4df) __A,
        (__v4df) __W,
        (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sqrt_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_sqrtpd256_mask ((__v4df) __A,
        (__v4df)
        _mm256_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sqrt_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_sqrtpd128_mask ((__v2df) __A,
        (__v2df) __W,
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sqrt_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_sqrtpd128_mask ((__v2df) __A,
        (__v2df)
        _mm_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sqrt_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_sqrtps256_mask ((__v8sf) __A,
       (__v8sf) __W,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sqrt_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_sqrtps256_mask ((__v8sf) __A,
       (__v8sf)
       _mm256_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sqrt_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_sqrtps128_mask ((__v4sf) __A,
       (__v4sf) __W,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sqrt_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_sqrtps128_mask ((__v4sf) __A,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_paddd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_paddq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_psubd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_psubq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_paddd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_paddq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psubd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psubq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_getexp_ps (__m256 __A)
{
  return (__m256) __builtin_ia32_getexpps256_mask ((__v8sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_getexp_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_getexpps256_mask ((__v8sf) __A,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_getexp_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_getexpps256_mask ((__v8sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_getexp_pd (__m256d __A)
{
  return (__m256d) __builtin_ia32_getexppd256_mask ((__v4df) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_getexp_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_getexppd256_mask ((__v4df) __A,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_getexp_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_getexppd256_mask ((__v4df) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_getexp_ps (__m128 __A)
{
  return (__m128) __builtin_ia32_getexpps128_mask ((__v4sf) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_getexp_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_getexpps128_mask ((__v4sf) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_getexp_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_getexpps128_mask ((__v4sf) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_getexp_pd (__m128d __A)
{
  return (__m128d) __builtin_ia32_getexppd128_mask ((__v2df) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_getexp_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_getexppd128_mask ((__v2df) __A,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_getexp_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_getexppd128_mask ((__v2df) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srl_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psrld256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srl_epi32 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psrld256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srl_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psrld128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srl_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrld128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srl_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psrlq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srl_epi64 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psrlq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srl_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srl_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_and_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pandd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_and_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pandd256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_scalef_pd (__m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_scalefpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_scalef_pd (__m256d __W, __mmask8 __U, __m256d __A,
         __m256d __B)
{
  return (__m256d) __builtin_ia32_scalefpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_scalef_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_scalefpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_scalef_ps (__m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_scalefps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_scalef_ps (__m256 __W, __mmask8 __U, __m256 __A,
         __m256 __B)
{
  return (__m256) __builtin_ia32_scalefps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_scalef_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_scalefps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_scalef_pd (__m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_scalefpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_scalef_pd (__m128d __W, __mmask8 __U, __m128d __A,
      __m128d __B)
{
  return (__m128d) __builtin_ia32_scalefpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_scalef_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_scalefpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_scalef_ps (__m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_scalefps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_scalef_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_scalefps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_scalef_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_scalefps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmadd_pd (__m256d __A, __mmask8 __U, __m256d __B,
        __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df) __C,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmadd_pd (__m256d __A, __m256d __B, __m256d __C,
         __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_mask3 ((__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmadd_pd (__mmask8 __U, __m256d __A, __m256d __B,
         __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_maskz ((__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmadd_pd (__m128d __A, __mmask8 __U, __m128d __B, __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df) __C,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmadd_pd (__m128d __A, __m128d __B, __m128d __C,
      __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_mask3 ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmadd_pd (__mmask8 __U, __m128d __A, __m128d __B,
      __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_maskz ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmadd_ps (__m256 __A, __mmask8 __U, __m256 __B, __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf) __C,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmadd_ps (__m256 __A, __m256 __B, __m256 __C,
         __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfmaddps256_mask3 ((__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmadd_ps (__mmask8 __U, __m256 __A, __m256 __B,
         __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_maskz ((__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmadd_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf) __C,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmadd_ps (__m128 __A, __m128 __B, __m128 __C, __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfmaddps128_mask3 ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmadd_ps (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_maskz ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmsub_pd (__m256d __A, __mmask8 __U, __m256d __B,
        __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_mask ((__v4df) __A,
          (__v4df) __B,
          -(__v4df) __C,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmsub_pd (__m256d __A, __m256d __B, __m256d __C,
         __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfmsubpd256_mask3 ((__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmsub_pd (__mmask8 __U, __m256d __A, __m256d __B,
         __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_maskz ((__v4df) __A,
           (__v4df) __B,
           -(__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmsub_pd (__m128d __A, __mmask8 __U, __m128d __B, __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_mask ((__v2df) __A,
          (__v2df) __B,
          -(__v2df) __C,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmsub_pd (__m128d __A, __m128d __B, __m128d __C,
      __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfmsubpd128_mask3 ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmsub_pd (__mmask8 __U, __m128d __A, __m128d __B,
      __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_maskz ((__v2df) __A,
           (__v2df) __B,
           -(__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmsub_ps (__m256 __A, __mmask8 __U, __m256 __B, __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         -(__v8sf) __C,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmsub_ps (__m256 __A, __m256 __B, __m256 __C,
         __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfmsubps256_mask3 ((__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmsub_ps (__mmask8 __U, __m256 __A, __m256 __B,
         __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_maskz ((__v8sf) __A,
          (__v8sf) __B,
          -(__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmsub_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         -(__v4sf) __C,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmsub_ps (__m128 __A, __m128 __B, __m128 __C, __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfmsubps128_mask3 ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmsub_ps (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_maskz ((__v4sf) __A,
          (__v4sf) __B,
          -(__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmaddsub_pd (__m256d __A, __mmask8 __U, __m256d __B,
    __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddsubpd256_mask ((__v4df) __A,
             (__v4df) __B,
             (__v4df) __C,
             (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmaddsub_pd (__m256d __A, __m256d __B, __m256d __C,
     __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfmaddsubpd256_mask3 ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __C,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmaddsub_pd (__mmask8 __U, __m256d __A, __m256d __B,
     __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddsubpd256_maskz ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __C,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmaddsub_pd (__m128d __A, __mmask8 __U, __m128d __B,
        __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddsubpd128_mask ((__v2df) __A,
             (__v2df) __B,
             (__v2df) __C,
             (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmaddsub_pd (__m128d __A, __m128d __B, __m128d __C,
         __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfmaddsubpd128_mask3 ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __C,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmaddsub_pd (__mmask8 __U, __m128d __A, __m128d __B,
         __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddsubpd128_maskz ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __C,
       (__mmask8)
       __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmaddsub_ps (__m256 __A, __mmask8 __U, __m256 __B,
    __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddsubps256_mask ((__v8sf) __A,
            (__v8sf) __B,
            (__v8sf) __C,
            (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmaddsub_ps (__m256 __A, __m256 __B, __m256 __C,
     __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfmaddsubps256_mask3 ((__v8sf) __A,
             (__v8sf) __B,
             (__v8sf) __C,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmaddsub_ps (__mmask8 __U, __m256 __A, __m256 __B,
     __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddsubps256_maskz ((__v8sf) __A,
             (__v8sf) __B,
             (__v8sf) __C,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmaddsub_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddsubps128_mask ((__v4sf) __A,
            (__v4sf) __B,
            (__v4sf) __C,
            (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmaddsub_ps (__m128 __A, __m128 __B, __m128 __C,
         __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfmaddsubps128_mask3 ((__v4sf) __A,
             (__v4sf) __B,
             (__v4sf) __C,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmaddsub_ps (__mmask8 __U, __m128 __A, __m128 __B,
         __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddsubps128_maskz ((__v4sf) __A,
             (__v4sf) __B,
             (__v4sf) __C,
             (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmsubadd_pd (__m256d __A, __mmask8 __U, __m256d __B,
    __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddsubpd256_mask ((__v4df) __A,
             (__v4df) __B,
             -(__v4df) __C,
             (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmsubadd_pd (__m256d __A, __m256d __B, __m256d __C,
     __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfmsubaddpd256_mask3 ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __C,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmsubadd_pd (__mmask8 __U, __m256d __A, __m256d __B,
     __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddsubpd256_maskz ((__v4df) __A,
       (__v4df) __B,
       -(__v4df) __C,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmsubadd_pd (__m128d __A, __mmask8 __U, __m128d __B,
        __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddsubpd128_mask ((__v2df) __A,
             (__v2df) __B,
             -(__v2df) __C,
             (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmsubadd_pd (__m128d __A, __m128d __B, __m128d __C,
         __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfmsubaddpd128_mask3 ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __C,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmsubadd_pd (__mmask8 __U, __m128d __A, __m128d __B,
         __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddsubpd128_maskz ((__v2df) __A,
       (__v2df) __B,
       -(__v2df) __C,
       (__mmask8)
       __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fmsubadd_ps (__m256 __A, __mmask8 __U, __m256 __B,
    __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddsubps256_mask ((__v8sf) __A,
            (__v8sf) __B,
            -(__v8sf) __C,
            (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fmsubadd_ps (__m256 __A, __m256 __B, __m256 __C,
     __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfmsubaddps256_mask3 ((__v8sf) __A,
             (__v8sf) __B,
             (__v8sf) __C,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fmsubadd_ps (__mmask8 __U, __m256 __A, __m256 __B,
     __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddsubps256_maskz ((__v8sf) __A,
             (__v8sf) __B,
             -(__v8sf) __C,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fmsubadd_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddsubps128_mask ((__v4sf) __A,
            (__v4sf) __B,
            -(__v4sf) __C,
            (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fmsubadd_ps (__m128 __A, __m128 __B, __m128 __C,
         __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfmsubaddps128_mask3 ((__v4sf) __A,
             (__v4sf) __B,
             (__v4sf) __C,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fmsubadd_ps (__mmask8 __U, __m128 __A, __m128 __B,
         __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddsubps128_maskz ((__v4sf) __A,
             (__v4sf) __B,
             -(__v4sf) __C,
             (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fnmadd_pd (__m256d __A, __mmask8 __U, __m256d __B,
         __m256d __C)
{
  return (__m256d) __builtin_ia32_vfnmaddpd256_mask ((__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fnmadd_pd (__m256d __A, __m256d __B, __m256d __C,
   __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_mask3 (-(__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fnmadd_pd (__mmask8 __U, __m256d __A, __m256d __B,
   __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_maskz (-(__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fnmadd_pd (__m128d __A, __mmask8 __U, __m128d __B,
      __m128d __C)
{
  return (__m128d) __builtin_ia32_vfnmaddpd128_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fnmadd_pd (__m128d __A, __m128d __B, __m128d __C,
       __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_mask3 (-(__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fnmadd_pd (__mmask8 __U, __m128d __A, __m128d __B,
       __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_maskz (-(__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fnmadd_ps (__m256 __A, __mmask8 __U, __m256 __B,
         __m256 __C)
{
  return (__m256) __builtin_ia32_vfnmaddps256_mask ((__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fnmadd_ps (__m256 __A, __m256 __B, __m256 __C,
   __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfmaddps256_mask3 (-(__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fnmadd_ps (__mmask8 __U, __m256 __A, __m256 __B,
   __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_maskz (-(__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fnmadd_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfnmaddps128_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fnmadd_ps (__m128 __A, __m128 __B, __m128 __C, __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfmaddps128_mask3 (-(__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fnmadd_ps (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_maskz (-(__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fnmsub_pd (__m256d __A, __mmask8 __U, __m256d __B,
         __m256d __C)
{
  return (__m256d) __builtin_ia32_vfnmsubpd256_mask ((__v4df) __A,
           (__v4df) __B,
           (__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fnmsub_pd (__m256d __A, __m256d __B, __m256d __C,
   __mmask8 __U)
{
  return (__m256d) __builtin_ia32_vfnmsubpd256_mask3 ((__v4df) __A,
            (__v4df) __B,
            (__v4df) __C,
            (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fnmsub_pd (__mmask8 __U, __m256d __A, __m256d __B,
   __m256d __C)
{
  return (__m256d) __builtin_ia32_vfmaddpd256_maskz (-(__v4df) __A,
           (__v4df) __B,
           -(__v4df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fnmsub_pd (__m128d __A, __mmask8 __U, __m128d __B,
      __m128d __C)
{
  return (__m128d) __builtin_ia32_vfnmsubpd128_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fnmsub_pd (__m128d __A, __m128d __B, __m128d __C,
       __mmask8 __U)
{
  return (__m128d) __builtin_ia32_vfnmsubpd128_mask3 ((__v2df) __A,
            (__v2df) __B,
            (__v2df) __C,
            (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fnmsub_pd (__mmask8 __U, __m128d __A, __m128d __B,
       __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddpd128_maskz (-(__v2df) __A,
           (__v2df) __B,
           -(__v2df) __C,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_fnmsub_ps (__m256 __A, __mmask8 __U, __m256 __B,
         __m256 __C)
{
  return (__m256) __builtin_ia32_vfnmsubps256_mask ((__v8sf) __A,
          (__v8sf) __B,
          (__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask3_fnmsub_ps (__m256 __A, __m256 __B, __m256 __C,
   __mmask8 __U)
{
  return (__m256) __builtin_ia32_vfnmsubps256_mask3 ((__v8sf) __A,
           (__v8sf) __B,
           (__v8sf) __C,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_fnmsub_ps (__mmask8 __U, __m256 __A, __m256 __B,
   __m256 __C)
{
  return (__m256) __builtin_ia32_vfmaddps256_maskz (-(__v8sf) __A,
          (__v8sf) __B,
          -(__v8sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_fnmsub_ps (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfnmsubps128_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask3_fnmsub_ps (__m128 __A, __m128 __B, __m128 __C, __mmask8 __U)
{
  return (__m128) __builtin_ia32_vfnmsubps128_mask3 ((__v4sf) __A,
           (__v4sf) __B,
           (__v4sf) __C,
           (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_fnmsub_ps (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddps128_maskz (-(__v4sf) __A,
          (__v4sf) __B,
          -(__v4sf) __C,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_and_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pandd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_and_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pandd128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_andnot_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
     __m256i __B)
{
  return (__m256i) __builtin_ia32_pandnd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_andnot_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pandnd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_andnot_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_pandnd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_andnot_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pandnd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_or_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pord256_mask ((__v8si) __A,
      (__v8si) __B,
      (__v8si) __W,
      (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_or_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pord256_mask ((__v8si) __A,
      (__v8si) __B,
      (__v8si)
      _mm256_setzero_si256 (),
      (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_or_epi32 (__m128i __W, __mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pord128_mask ((__v4si) __A,
      (__v4si) __B,
      (__v4si) __W,
      (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_or_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pord128_mask ((__v4si) __A,
      (__v4si) __B,
      (__v4si)
      _mm_setzero_si128 (),
      (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_xor_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pxord256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_xor_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pxord256_mask ((__v8si) __A,
       (__v8si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_xor_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pxord128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_xor_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pxord128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtpd_ps (__m128 __W, __mmask8 __U, __m128d __A)
{
  return (__m128) __builtin_ia32_cvtpd2ps_mask ((__v2df) __A,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtpd_ps (__mmask8 __U, __m128d __A)
{
  return (__m128) __builtin_ia32_cvtpd2ps_mask ((__v2df) __A,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtpd_ps (__m128 __W, __mmask8 __U, __m256d __A)
{
  return (__m128) __builtin_ia32_cvtpd2ps256_mask ((__v4df) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtpd_ps (__mmask8 __U, __m256d __A)
{
  return (__m128) __builtin_ia32_cvtpd2ps256_mask ((__v4df) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtps_epi32 (__m256i __W, __mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvtps2dq256_mask ((__v8sf) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtps_epi32 (__mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvtps2dq256_mask ((__v8sf) __A,
          (__v8si)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtps_epi32 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2dq128_mask ((__v4sf) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtps_epi32 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2dq128_mask ((__v4sf) __A,
          (__v4si)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtps_epu32 (__m256 __A)
{
  return (__m256i) __builtin_ia32_cvtps2udq256_mask ((__v8sf) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtps_epu32 (__m256i __W, __mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvtps2udq256_mask ((__v8sf) __A,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtps_epu32 (__mmask8 __U, __m256 __A)
{
  return (__m256i) __builtin_ia32_cvtps2udq256_mask ((__v8sf) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_epu32 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2udq128_mask ((__v4sf) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtps_epu32 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2udq128_mask ((__v4sf) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtps_epu32 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2udq128_mask ((__v4sf) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_movedup_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_movddup256_mask ((__v4df) __A,
         (__v4df) __W,
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_movedup_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_movddup256_mask ((__v4df) __A,
         (__v4df)
         _mm256_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_movedup_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_movddup128_mask ((__v2df) __A,
         (__v2df) __W,
         (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_movedup_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_movddup128_mask ((__v2df) __A,
         (__v2df)
         _mm_setzero_pd (),
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_movehdup_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movshdup256_mask ((__v8sf) __A,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_movehdup_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movshdup256_mask ((__v8sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_movehdup_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movshdup128_mask ((__v4sf) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_movehdup_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movshdup128_mask ((__v4sf) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_moveldup_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movsldup256_mask ((__v8sf) __A,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_moveldup_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_movsldup256_mask ((__v8sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_moveldup_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movsldup128_mask ((__v4sf) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_moveldup_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_movsldup128_mask ((__v4sf) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhdq128_mask ((__v4si) __A,
           (__v4si) __B,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhdq128_mask ((__v4si) __A,
           (__v4si) __B,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhdq256_mask ((__v8si) __A,
           (__v8si) __B,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhdq256_mask ((__v8si) __A,
           (__v8si) __B,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhqdq128_mask ((__v2di) __A,
            (__v2di) __B,
            (__v2di) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhqdq128_mask ((__v2di) __A,
            (__v2di) __B,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhqdq256_mask ((__v4di) __A,
            (__v4di) __B,
            (__v4di) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhqdq256_mask ((__v4di) __A,
            (__v4di) __B,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckldq128_mask ((__v4si) __A,
           (__v4si) __B,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckldq128_mask ((__v4si) __A,
           (__v4si) __B,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckldq256_mask ((__v8si) __A,
           (__v8si) __B,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckldq256_mask ((__v8si) __A,
           (__v8si) __B,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklqdq128_mask ((__v2di) __A,
            (__v2di) __B,
            (__v2di) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklqdq128_mask ((__v2di) __A,
            (__v2di) __B,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklqdq256_mask ((__v4di) __A,
            (__v4di) __B,
            (__v4di) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklqdq256_mask ((__v4di) __A,
            (__v4di) __B,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epu32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __A,
         (__v4si) __B, 0,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqd128_mask ((__v4si) __A,
          (__v4si) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epu32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __A,
         (__v4si) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epi32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqd128_mask ((__v4si) __A,
          (__v4si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epu32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __A,
         (__v8si) __B, 0,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqd256_mask ((__v8si) __A,
          (__v8si) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epu32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __A,
         (__v8si) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epi32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqd256_mask ((__v8si) __A,
          (__v8si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epu64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __A,
         (__v2di) __B, 0,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq128_mask ((__v2di) __A,
          (__v2di) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epu64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __A,
         (__v2di) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epi64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq128_mask ((__v2di) __A,
          (__v2di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epu64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __A,
         (__v4di) __B, 0,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq256_mask ((__v4di) __A,
          (__v4di) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epu64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __A,
         (__v4di) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epi64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqq256_mask ((__v4di) __A,
          (__v4di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epu32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __A,
         (__v4si) __B, 6,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtd128_mask ((__v4si) __A,
          (__v4si) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epu32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __A,
         (__v4si) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epi32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtd128_mask ((__v4si) __A,
          (__v4si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epu32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __A,
         (__v8si) __B, 6,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtd256_mask ((__v8si) __A,
          (__v8si) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epu32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __A,
         (__v8si) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epi32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtd256_mask ((__v8si) __A,
          (__v8si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epu64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __A,
         (__v2di) __B, 6,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq128_mask ((__v2di) __A,
          (__v2di) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epu64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __A,
         (__v2di) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epi64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq128_mask ((__v2di) __A,
          (__v2di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epu64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __A,
         (__v4di) __B, 6,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq256_mask ((__v4di) __A,
          (__v4di) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epu64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __A,
         (__v4di) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epi64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtq256_mask ((__v4di) __A,
          (__v4di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_test_epi32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmd128 ((__v4si) __A,
            (__v4si) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_test_epi32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmd128 ((__v4si) __A,
            (__v4si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_test_epi32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestmd256 ((__v8si) __A,
            (__v8si) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_test_epi32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestmd256 ((__v8si) __A,
            (__v8si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_test_epi64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq128 ((__v2di) __A,
            (__v2di) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_test_epi64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq128 ((__v2di) __A,
            (__v2di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_test_epi64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq256 ((__v4di) __A,
            (__v4di) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_test_epi64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestmq256 ((__v4di) __A,
            (__v4di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_testn_epi32_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmd128 ((__v4si) __A,
      (__v4si) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_testn_epi32_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmd128 ((__v4si) __A,
      (__v4si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testn_epi32_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmd256 ((__v8si) __A,
      (__v8si) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_testn_epi32_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmd256 ((__v8si) __A,
      (__v8si) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_testn_epi64_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq128 ((__v2di) __A,
      (__v2di) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_testn_epi64_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq128 ((__v2di) __A,
      (__v2di) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testn_epi64_mask (__m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq256 ((__v4di) __A,
      (__v4di) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_testn_epi64_mask (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmq256 ((__v4di) __A,
      (__v4di) __B, __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_compressdf256_mask ((__v4df) __A,
            (__v4df) __W,
            (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_compressdf256_mask ((__v4df) __A,
            (__v4df)
            _mm256_setzero_pd (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_pd (void *__P, __mmask8 __U, __m256d __A)
{
  __builtin_ia32_compressstoredf256_mask ((__v4df *) __P,
       (__v4df) __A,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_compressdf128_mask ((__v2df) __A,
            (__v2df) __W,
            (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_compressdf128_mask ((__v2df) __A,
            (__v2df)
            _mm_setzero_pd (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_pd (void *__P, __mmask8 __U, __m128d __A)
{
  __builtin_ia32_compressstoredf128_mask ((__v2df *) __P,
       (__v2df) __A,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_compresssf256_mask ((__v8sf) __A,
           (__v8sf) __W,
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_compresssf256_mask ((__v8sf) __A,
           (__v8sf)
           _mm256_setzero_ps (),
           (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_ps (void *__P, __mmask8 __U, __m256 __A)
{
  __builtin_ia32_compressstoresf256_mask ((__v8sf *) __P,
       (__v8sf) __A,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_compresssf128_mask ((__v4sf) __A,
           (__v4sf) __W,
           (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_compresssf128_mask ((__v4sf) __A,
           (__v4sf)
           _mm_setzero_ps (),
           (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_ps (void *__P, __mmask8 __U, __m128 __A)
{
  __builtin_ia32_compressstoresf128_mask ((__v4sf *) __P,
       (__v4sf) __A,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_compressdi256_mask ((__v4di) __A,
            (__v4di) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_compressdi256_mask ((__v4di) __A,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_epi64 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_compressstoredi256_mask ((__v4di *) __P,
       (__v4di) __A,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_compressdi128_mask ((__v2di) __A,
            (__v2di) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_compressdi128_mask ((__v2di) __A,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_epi64 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_compressstoredi128_mask ((__v2di *) __P,
       (__v2di) __A,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_compresssi256_mask ((__v8si) __A,
            (__v8si) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_compresssi256_mask ((__v8si) __A,
            (__v8si)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_epi32 (void *__P, __mmask8 __U, __m256i __A)
{
  __builtin_ia32_compressstoresi256_mask ((__v8si *) __P,
       (__v8si) __A,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_compresssi128_mask ((__v4si) __A,
            (__v4si) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_compresssi128_mask ((__v4si) __A,
            (__v4si)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_epi32 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_compressstoresi128_mask ((__v4si *) __P,
       (__v4si) __A,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_pd (__m256d __W, __mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_expanddf256_mask ((__v4df) __A,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_pd (__mmask8 __U, __m256d __A)
{
  return (__m256d) __builtin_ia32_expanddf256_maskz ((__v4df) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_pd (__m256d __W, __mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_expandloaddf256_mask ((__v4df *) __P,
       (__v4df) __W,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_pd (__mmask8 __U, void const *__P)
{
  return (__m256d) __builtin_ia32_expandloaddf256_maskz ((__v4df *) __P,
        (__v4df)
        _mm256_setzero_pd (),
        (__mmask8)
        __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_pd (__m128d __W, __mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_expanddf128_mask ((__v2df) __A,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_pd (__mmask8 __U, __m128d __A)
{
  return (__m128d) __builtin_ia32_expanddf128_maskz ((__v2df) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_pd (__m128d __W, __mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_expandloaddf128_mask ((__v2df *) __P,
       (__v2df) __W,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_pd (__mmask8 __U, void const *__P)
{
  return (__m128d) __builtin_ia32_expandloaddf128_maskz ((__v2df *) __P,
        (__v2df)
        _mm_setzero_pd (),
        (__mmask8)
        __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_ps (__m256 __W, __mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_expandsf256_mask ((__v8sf) __A,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_ps (__mmask8 __U, __m256 __A)
{
  return (__m256) __builtin_ia32_expandsf256_maskz ((__v8sf) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_ps (__m256 __W, __mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_expandloadsf256_mask ((__v8sf *) __P,
             (__v8sf) __W,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_ps (__mmask8 __U, void const *__P)
{
  return (__m256) __builtin_ia32_expandloadsf256_maskz ((__v8sf *) __P,
       (__v8sf)
       _mm256_setzero_ps (),
       (__mmask8)
       __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_ps (__m128 __W, __mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_expandsf128_mask ((__v4sf) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_ps (__mmask8 __U, __m128 __A)
{
  return (__m128) __builtin_ia32_expandsf128_maskz ((__v4sf) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_ps (__m128 __W, __mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_expandloadsf128_mask ((__v4sf *) __P,
             (__v4sf) __W,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_ps (__mmask8 __U, void const *__P)
{
  return (__m128) __builtin_ia32_expandloadsf128_maskz ((__v4sf *) __P,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_expanddi256_mask ((__v4di) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_expanddi256_maskz ((__v4di) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_epi64 (__m256i __W, __mmask8 __U,
          void const *__P)
{
  return (__m256i) __builtin_ia32_expandloaddi256_mask ((__v4di *) __P,
       (__v4di) __W,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_expandloaddi256_maskz ((__v4di *) __P,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_expanddi128_mask ((__v2di) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_expanddi128_maskz ((__v2di) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_epi64 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_expandloaddi128_mask ((__v2di *) __P,
       (__v2di) __W,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_epi64 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_expandloaddi128_maskz ((__v2di *) __P,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8)
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_expandsi256_mask ((__v8si) __A,
          (__v8si) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_expandsi256_maskz ((__v8si) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_epi32 (__m256i __W, __mmask8 __U,
          void const *__P)
{
  return (__m256i) __builtin_ia32_expandloadsi256_mask ((__v8si *) __P,
       (__v8si) __W,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_epi32 (__mmask8 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_expandloadsi256_maskz ((__v8si *) __P,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_expandsi128_mask ((__v4si) __A,
          (__v4si) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_expandsi128_maskz ((__v4si) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_epi32 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_expandloadsi128_mask ((__v4si *) __P,
       (__v4si) __W,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_epi32 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_expandloadsi128_maskz ((__v4si *) __P,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8)
        __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_pd (__m256d __A, __m256i __I, __m256d __B)
{
  return (__m256d) __builtin_ia32_vpermt2varpd256_mask ((__v4di) __I
                 ,
       (__v4df) __A,
       (__v4df) __B,
       (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_pd (__m256d __A, __mmask8 __U, __m256i __I,
        __m256d __B)
{
  return (__m256d) __builtin_ia32_vpermt2varpd256_mask ((__v4di) __I
                 ,
       (__v4df) __A,
       (__v4df) __B,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_pd (__m256d __A, __m256i __I, __mmask8 __U,
         __m256d __B)
{
  return (__m256d) __builtin_ia32_vpermi2varpd256_mask ((__v4df) __A,
       (__v4di) __I
                 ,
       (__v4df) __B,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_pd (__mmask8 __U, __m256d __A, __m256i __I,
         __m256d __B)
{
  return (__m256d) __builtin_ia32_vpermt2varpd256_maskz ((__v4di) __I
                  ,
        (__v4df) __A,
        (__v4df) __B,
        (__mmask8)
        __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_ps (__m256 __A, __m256i __I, __m256 __B)
{
  return (__m256) __builtin_ia32_vpermt2varps256_mask ((__v8si) __I
                       ,
             (__v8sf) __A,
             (__v8sf) __B,
             (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_ps (__m256 __A, __mmask8 __U, __m256i __I,
        __m256 __B)
{
  return (__m256) __builtin_ia32_vpermt2varps256_mask ((__v8si) __I
                       ,
             (__v8sf) __A,
             (__v8sf) __B,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_ps (__m256 __A, __m256i __I, __mmask8 __U,
         __m256 __B)
{
  return (__m256) __builtin_ia32_vpermi2varps256_mask ((__v8sf) __A,
             (__v8si) __I
                       ,
             (__v8sf) __B,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_ps (__mmask8 __U, __m256 __A, __m256i __I,
         __m256 __B)
{
  return (__m256) __builtin_ia32_vpermt2varps256_maskz ((__v8si) __I
                 ,
       (__v8sf) __A,
       (__v8sf) __B,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_epi64 (__m128i __A, __m128i __I, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varq128_mask ((__v2di) __I
                       ,
             (__v2di) __A,
             (__v2di) __B,
             (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_epi64 (__m128i __A, __mmask8 __U, __m128i __I,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varq128_mask ((__v2di) __I
                       ,
             (__v2di) __A,
             (__v2di) __B,
             (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_epi64 (__m128i __A, __m128i __I, __mmask8 __U,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermi2varq128_mask ((__v2di) __A,
             (__v2di) __I
                       ,
             (__v2di) __B,
             (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_epi64 (__mmask8 __U, __m128i __A, __m128i __I,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varq128_maskz ((__v2di) __I
                 ,
       (__v2di) __A,
       (__v2di) __B,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_epi32 (__m128i __A, __m128i __I, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2vard128_mask ((__v4si) __I
                       ,
             (__v4si) __A,
             (__v4si) __B,
             (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_epi32 (__m128i __A, __mmask8 __U, __m128i __I,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2vard128_mask ((__v4si) __I
                       ,
             (__v4si) __A,
             (__v4si) __B,
             (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_epi32 (__m128i __A, __m128i __I, __mmask8 __U,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermi2vard128_mask ((__v4si) __A,
             (__v4si) __I
                       ,
             (__v4si) __B,
             (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_epi32 (__mmask8 __U, __m128i __A, __m128i __I,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2vard128_maskz ((__v4si) __I
                 ,
       (__v4si) __A,
       (__v4si) __B,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_epi64 (__m256i __A, __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varq256_mask ((__v4di) __I
                       ,
             (__v4di) __A,
             (__v4di) __B,
             (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_epi64 (__m256i __A, __mmask8 __U, __m256i __I,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varq256_mask ((__v4di) __I
                       ,
             (__v4di) __A,
             (__v4di) __B,
             (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_epi64 (__m256i __A, __m256i __I,
     __mmask8 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermi2varq256_mask ((__v4di) __A,
             (__v4di) __I
                       ,
             (__v4di) __B,
             (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_epi64 (__mmask8 __U, __m256i __A,
     __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varq256_maskz ((__v4di) __I
                 ,
       (__v4di) __A,
       (__v4di) __B,
       (__mmask8)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_epi32 (__m256i __A, __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2vard256_mask ((__v8si) __I
                       ,
             (__v8si) __A,
             (__v8si) __B,
             (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_epi32 (__m256i __A, __mmask8 __U, __m256i __I,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2vard256_mask ((__v8si) __I
                       ,
             (__v8si) __A,
             (__v8si) __B,
             (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_epi32 (__m256i __A, __m256i __I,
     __mmask8 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermi2vard256_mask ((__v8si) __A,
             (__v8si) __I
                       ,
             (__v8si) __B,
             (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_epi32 (__mmask8 __U, __m256i __A,
     __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2vard256_maskz ((__v8si) __I
                 ,
       (__v8si) __A,
       (__v8si) __B,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_pd (__m128d __A, __m128i __I, __m128d __B)
{
  return (__m128d) __builtin_ia32_vpermt2varpd128_mask ((__v2di) __I
                 ,
       (__v2df) __A,
       (__v2df) __B,
       (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_pd (__m128d __A, __mmask8 __U, __m128i __I,
     __m128d __B)
{
  return (__m128d) __builtin_ia32_vpermt2varpd128_mask ((__v2di) __I
                 ,
       (__v2df) __A,
       (__v2df) __B,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_pd (__m128d __A, __m128i __I, __mmask8 __U,
      __m128d __B)
{
  return (__m128d) __builtin_ia32_vpermi2varpd128_mask ((__v2df) __A,
       (__v2di) __I
                 ,
       (__v2df) __B,
       (__mmask8)
       __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_pd (__mmask8 __U, __m128d __A, __m128i __I,
      __m128d __B)
{
  return (__m128d) __builtin_ia32_vpermt2varpd128_maskz ((__v2di) __I
                  ,
        (__v2df) __A,
        (__v2df) __B,
        (__mmask8)
        __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_ps (__m128 __A, __m128i __I, __m128 __B)
{
  return (__m128) __builtin_ia32_vpermt2varps128_mask ((__v4si) __I
                       ,
             (__v4sf) __A,
             (__v4sf) __B,
             (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_ps (__m128 __A, __mmask8 __U, __m128i __I,
     __m128 __B)
{
  return (__m128) __builtin_ia32_vpermt2varps128_mask ((__v4si) __I
                       ,
             (__v4sf) __A,
             (__v4sf) __B,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_ps (__m128 __A, __m128i __I, __mmask8 __U,
      __m128 __B)
{
  return (__m128) __builtin_ia32_vpermi2varps128_mask ((__v4sf) __A,
             (__v4si) __I
                       ,
             (__v4sf) __B,
             (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_ps (__mmask8 __U, __m128 __A, __m128i __I,
      __m128 __B)
{
  return (__m128) __builtin_ia32_vpermt2varps128_maskz ((__v4si) __I
                 ,
       (__v4sf) __A,
       (__v4sf) __B,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srav_epi64 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psravq128_mask ((__v2di) __X,
        (__v2di) __Y,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srav_epi64 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psravq128_mask ((__v2di) __X,
        (__v2di) __Y,
        (__v2di) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srav_epi64 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psravq128_mask ((__v2di) __X,
        (__v2di) __Y,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sllv_epi32 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sllv_epi32 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sllv_epi32 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sllv_epi32 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sllv_epi64 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv4di_mask ((__v4di) __X,
       (__v4di) __Y,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sllv_epi64 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psllv4di_mask ((__v4di) __X,
       (__v4di) __Y,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sllv_epi64 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv2di_mask ((__v2di) __X,
       (__v2di) __Y,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sllv_epi64 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psllv2di_mask ((__v2di) __X,
       (__v2di) __Y,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srav_epi32 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrav8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srav_epi32 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrav8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srav_epi32 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrav4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srav_epi32 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrav4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srlv_epi32 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srlv_epi32 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv8si_mask ((__v8si) __X,
       (__v8si) __Y,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srlv_epi32 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srlv_epi32 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv4si_mask ((__v4si) __X,
       (__v4si) __Y,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srlv_epi64 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv4di_mask ((__v4di) __X,
       (__v4di) __Y,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srlv_epi64 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psrlv4di_mask ((__v4di) __X,
       (__v4di) __Y,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srlv_epi64 (__m128i __W, __mmask8 __U, __m128i __X,
       __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv2di_mask ((__v2di) __X,
       (__v2di) __Y,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srlv_epi64 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_psrlv2di_mask ((__v2di) __X,
       (__v2di) __Y,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rolv_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rolv_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rolv_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rolv_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rolv_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rolv_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rorv_epi32 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rorv_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rorv_epi32 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rorv_epi32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rorv_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rorv_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rolv_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rolv_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rolv_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prolvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rolv_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rolv_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rolv_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prolvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_rorv_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_rorv_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_rorv_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_prorvq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_rorv_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_rorv_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_rorv_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_prorvq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srav_epi64 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psravq256_mask ((__v4di) __X,
        (__v4di) __Y,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srav_epi64 (__m256i __W, __mmask8 __U, __m256i __X,
   __m256i __Y)
{
  return (__m256i) __builtin_ia32_psravq256_mask ((__v4di) __X,
        (__v4di) __Y,
        (__v4di) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srav_epi64 (__mmask8 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_psravq256_mask ((__v4di) __X,
        (__v4di) __Y,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_and_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pandq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di) __W, __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_and_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pandq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di)
       _mm256_setzero_pd (),
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_and_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pandq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W, __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_and_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pandq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_pd (),
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_andnot_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
     __m256i __B)
{
  return (__m256i) __builtin_ia32_pandnq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W, __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_andnot_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pandnq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_pd (),
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_andnot_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_pandnq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W, __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_andnot_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pandnq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_pd (),
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_or_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_porq256_mask ((__v4di) __A,
      (__v4di) __B,
      (__v4di) __W,
      (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_or_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_porq256_mask ((__v4di) __A,
      (__v4di) __B,
      (__v4di)
      _mm256_setzero_si256 (),
      (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_or_epi64 (__m128i __W, __mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_porq128_mask ((__v2di) __A,
      (__v2di) __B,
      (__v2di) __W,
      (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_or_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_porq128_mask ((__v2di) __A,
      (__v2di) __B,
      (__v2di)
      _mm_setzero_si128 (),
      (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_xor_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pxorq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_xor_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pxorq256_mask ((__v4di) __A,
       (__v4di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_xor_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pxorq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_xor_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pxorq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_maxpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_maxpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_maxps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_maxps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_div_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_divps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_div_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_divps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_div_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_divpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_div_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_divpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_minpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_div_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_divpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_minpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_minps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_div_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_divpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_div_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_divps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_minps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_div_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_divps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_mulps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_minps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_mulps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_maxps_mask ((__v4sf) __A,
          (__v4sf) __B,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_minpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_minpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_maxpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_maxpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_mulpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_mulpd_mask ((__v2df) __A,
           (__v2df) __B,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mul_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_mulps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mul_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_mulps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mul_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_mulpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mul_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_mulpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epi64 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epi64 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epi64 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epi64 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epu64 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_max_epu64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epu64 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_min_epu64 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epu64 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epu64 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminuq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epi32 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epi32 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epi32 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epi32 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsd256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epu32 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxud256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epu32 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxud256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epu32 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminud256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epu32 (__m256i __W, __mmask8 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminud256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epi64 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epi64 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epi64 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epi64 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epu64 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_max_epu64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epu64 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_min_epu64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epu64 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epu64 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminuq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epi32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epi32 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epi32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epi32 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsd128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epu32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxud128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epu32 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxud128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epu32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminud128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epu32 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminud128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W, __M);
}
#pragma GCC push_options
#pragma GCC target("avx512vl,avx512cd")
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastmb_epi64 (__mmask8 __A)
{
  return (__m128i) __builtin_ia32_broadcastmb128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastmb_epi64 (__mmask8 __A)
{
  return (__m256i) __builtin_ia32_broadcastmb256 (__A);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcastmw_epi32 (__mmask16 __A)
{
  return (__m128i) __builtin_ia32_broadcastmw128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcastmw_epi32 (__mmask16 __A)
{
  return (__m256i) __builtin_ia32_broadcastmw256 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_lzcnt_epi32 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntd_256_mask ((__v8si) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_lzcnt_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntd_256_mask ((__v8si) __A,
           (__v8si) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_lzcnt_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntd_256_mask ((__v8si) __A,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_lzcnt_epi64 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntq_256_mask ((__v4di) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_lzcnt_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntq_256_mask ((__v4di) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_lzcnt_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vplzcntq_256_mask ((__v4di) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_conflict_epi64 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictdi_256_mask ((__v4di) __A,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_conflict_epi64 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictdi_256_mask ((__v4di) __A,
        (__v4di) __W,
        (__mmask8)
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_conflict_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictdi_256_mask ((__v4di) __A,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8)
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_conflict_epi32 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictsi_256_mask ((__v8si) __A,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_conflict_epi32 (__m256i __W, __mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictsi_256_mask ((__v8si) __A,
        (__v8si) __W,
        (__mmask8)
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_conflict_epi32 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpconflictsi_256_mask ((__v8si) __A,
        (__v8si)
        _mm256_setzero_si256 (),
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_lzcnt_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntd_128_mask ((__v4si) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_lzcnt_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntd_128_mask ((__v4si) __A,
           (__v4si) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_lzcnt_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntd_128_mask ((__v4si) __A,
           (__v4si)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_lzcnt_epi64 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntq_128_mask ((__v2di) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_lzcnt_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntq_128_mask ((__v2di) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_lzcnt_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vplzcntq_128_mask ((__v2di) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_conflict_epi64 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictdi_128_mask ((__v2di) __A,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_conflict_epi64 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictdi_128_mask ((__v2di) __A,
        (__v2di) __W,
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_conflict_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictdi_128_mask ((__v2di) __A,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_conflict_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictsi_128_mask ((__v4si) __A,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_conflict_epi32 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictsi_128_mask ((__v4si) __A,
        (__v4si) __W,
        (__mmask8)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_conflict_epi32 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpconflictsi_128_mask ((__v4si) __A,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask8)
        __U);
}
#pragma GCC pop_options
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_pd (__m256d __W, __mmask8 __U, __m256d __A,
    __m256d __B)
{
  return (__m256d) __builtin_ia32_unpcklpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_unpcklpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_pd (__m128d __W, __mmask8 __U, __m128d __A,
        __m128d __B)
{
  return (__m128d) __builtin_ia32_unpcklpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_unpcklpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_ps (__m256 __W, __mmask8 __U, __m256 __A,
    __m256 __B)
{
  return (__m256) __builtin_ia32_unpcklps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_pd (__m256d __W, __mmask8 __U, __m256d __A,
    __m256d __B)
{
  return (__m256d) __builtin_ia32_unpckhpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_unpckhpd256_mask ((__v4df) __A,
          (__v4df) __B,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_pd (__m128d __W, __mmask8 __U, __m128d __A,
        __m128d __B)
{
  return (__m128d) __builtin_ia32_unpckhpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_unpckhpd128_mask ((__v2df) __A,
          (__v2df) __B,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_ps (__m256 __W, __mmask8 __U, __m256 __A,
    __m256 __B)
{
  return (__m256) __builtin_ia32_unpckhps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf) __W,
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_unpckhps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpckhps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpckhps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtph_ps (__m128 __W, __mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_vcvtph2ps_mask ((__v8hi) __A,
       (__v4sf) __W,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtph_ps (__mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_vcvtph2ps_mask ((__v8hi) __A,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_unpcklps256_mask ((__v8sf) __A,
         (__v8sf) __B,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtph_ps (__m256 __W, __mmask8 __U, __m128i __A)
{
  return (__m256) __builtin_ia32_vcvtph2ps256_mask ((__v8hi) __A,
          (__v8sf) __W,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtph_ps (__mmask8 __U, __m128i __A)
{
  return (__m256) __builtin_ia32_vcvtph2ps256_mask ((__v8hi) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpcklps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_unpcklps128_mask ((__v4sf) __A,
         (__v4sf) __B,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sra_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psrad256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sra_epi32 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psrad256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sra_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psrad128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sra_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrad128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sra_epi64 (__m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psraq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sra_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psraq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sra_epi64 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psraq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sra_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psraq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sra_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psraq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sra_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psraq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sll_epi32 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pslld128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sll_epi32 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pslld128_mask ((__v4si) __A,
       (__v4si) __B,
       (__v4si)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sll_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psllq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sll_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psllq128_mask ((__v2di) __A,
       (__v2di) __B,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sll_epi32 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_pslld256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sll_epi32 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_pslld256_mask ((__v8si) __A,
       (__v4si) __B,
       (__v8si)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sll_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psllq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di) __W,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sll_epi64 (__mmask8 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psllq256_mask ((__v4di) __A,
       (__v2di) __B,
       (__v4di)
       _mm256_setzero_si256 (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_ps (__m256 __W, __mmask8 __U, __m256i __X,
       __m256 __Y)
{
  return (__m256) __builtin_ia32_permvarsf256_mask ((__v8sf) __Y,
          (__v8si) __X,
          (__v8sf) __W,
          (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_ps (__mmask8 __U, __m256i __X, __m256 __Y)
{
  return (__m256) __builtin_ia32_permvarsf256_mask ((__v8sf) __Y,
          (__v8si) __X,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutexvar_pd (__m256i __X, __m256d __Y)
{
  return (__m256d) __builtin_ia32_permvardf256_mask ((__v4df) __Y,
           (__v4di) __X,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_pd (__m256d __W, __mmask8 __U, __m256i __X,
       __m256d __Y)
{
  return (__m256d) __builtin_ia32_permvardf256_mask ((__v4df) __Y,
           (__v4di) __X,
           (__v4df) __W,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_pd (__mmask8 __U, __m256i __X, __m256d __Y)
{
  return (__m256d) __builtin_ia32_permvardf256_mask ((__v4df) __Y,
           (__v4di) __X,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutevar_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256i __C)
{
  return (__m256d) __builtin_ia32_vpermilvarpd256_mask ((__v4df) __A,
       (__v4di) __C,
       (__v4df) __W,
       (__mmask8)
       __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutevar_pd (__mmask8 __U, __m256d __A, __m256i __C)
{
  return (__m256d) __builtin_ia32_vpermilvarpd256_mask ((__v4df) __A,
       (__v4di) __C,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8)
       __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutevar_ps (__m256 __W, __mmask8 __U, __m256 __A,
      __m256i __C)
{
  return (__m256) __builtin_ia32_vpermilvarps256_mask ((__v8sf) __A,
             (__v8si) __C,
             (__v8sf) __W,
             (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutevar_ps (__mmask8 __U, __m256 __A, __m256i __C)
{
  return (__m256) __builtin_ia32_vpermilvarps256_mask ((__v8sf) __A,
             (__v8si) __C,
             (__v8sf)
             _mm256_setzero_ps (),
             (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutevar_pd (__m128d __W, __mmask8 __U, __m128d __A,
   __m128i __C)
{
  return (__m128d) __builtin_ia32_vpermilvarpd_mask ((__v2df) __A,
           (__v2di) __C,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutevar_pd (__mmask8 __U, __m128d __A, __m128i __C)
{
  return (__m128d) __builtin_ia32_vpermilvarpd_mask ((__v2df) __A,
           (__v2di) __C,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutevar_ps (__m128 __W, __mmask8 __U, __m128 __A,
   __m128i __C)
{
  return (__m128) __builtin_ia32_vpermilvarps_mask ((__v4sf) __A,
          (__v4si) __C,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutevar_ps (__mmask8 __U, __m128 __A, __m128i __C)
{
  return (__m128) __builtin_ia32_vpermilvarps_mask ((__v4sf) __A,
          (__v4si) __C,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mullo_epi32 (__mmask8 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulld256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_epi64 (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvardi256_mask ((__v4di) __Y,
           (__v4di) __X,
           (__v4di)
           _mm256_setzero_si256 (),
           __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mullo_epi32 (__m256i __W, __mmask8 __M, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulld256_mask ((__v8si) __A,
        (__v8si) __B,
        (__v8si) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mullo_epi32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulld128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mullo_epi32 (__m128i __W, __mmask16 __M, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulld128_mask ((__v4si) __A,
        (__v4si) __B,
        (__v4si) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mul_epi32 (__m256i __W, __mmask8 __M, __m256i __X,
         __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmuldq256_mask ((__v8si) __X,
        (__v8si) __Y,
        (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mul_epi32 (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmuldq256_mask ((__v8si) __X,
        (__v8si) __Y,
        (__v4di)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_epi32 (__m128i __W, __mmask8 __M, __m128i __X,
      __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmuldq128_mask ((__v4si) __X,
        (__v4si) __Y,
        (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_epi32 (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmuldq128_mask ((__v4si) __X,
        (__v4si) __Y,
        (__v2di)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutexvar_epi64 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvardi256_mask ((__v4di) __Y,
           (__v4di) __X,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_epi64 (__m256i __W, __mmask8 __M, __m256i __X,
          __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvardi256_mask ((__v4di) __Y,
           (__v4di) __X,
           (__v4di) __W,
           __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mul_epu32 (__m256i __W, __mmask8 __M, __m256i __X,
         __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmuludq256_mask ((__v8si) __X,
         (__v8si) __Y,
         (__v4di) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_epi32 (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvarsi256_mask ((__v8si) __Y,
           (__v8si) __X,
           (__v8si)
           _mm256_setzero_si256 (),
           __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mul_epu32 (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmuludq256_mask ((__v8si) __X,
         (__v8si) __Y,
         (__v4di)
         _mm256_setzero_si256 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mul_epu32 (__m128i __W, __mmask8 __M, __m128i __X,
      __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmuludq128_mask ((__v4si) __X,
         (__v4si) __Y,
         (__v2di) __W, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mul_epu32 (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmuludq128_mask ((__v4si) __X,
         (__v4si) __Y,
         (__v2di)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutexvar_epi32 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvarsi256_mask ((__v8si) __Y,
           (__v8si) __X,
           (__v8si)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_epi32 (__m256i __W, __mmask8 __M, __m256i __X,
          __m256i __Y)
{
  return (__m256i) __builtin_ia32_permvarsi256_mask ((__v8si) __Y,
           (__v8si) __X,
           (__v8si) __W,
           __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epu32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 4,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epu32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 4,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epu32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 1,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epu32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 1,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epu32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 5,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epu32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 5,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epu32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 2,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epu32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd256_mask ((__v8si) __X,
        (__v8si) __Y, 2,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epu64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 4,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epu64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 4,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epu64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 1,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epu64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 1,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epu64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 5,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epu64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 5,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epu64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 2,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epu64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq256_mask ((__v4di) __X,
        (__v4di) __Y, 2,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epi32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 4,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epi32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 4,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epi32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 1,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epi32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 1,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epi32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 5,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epi32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 5,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epi32_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 2,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epi32_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd256_mask ((__v8si) __X,
       (__v8si) __Y, 2,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epi64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 4,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epi64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 4,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epi64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 1,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epi64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 1,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epi64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 5,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epi64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 5,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epi64_mask (__mmask8 __M, __m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 2,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epi64_mask (__m256i __X, __m256i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq256_mask ((__v4di) __X,
       (__v4di) __Y, 2,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epu32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 4,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epu32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 4,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epu32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 1,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epu32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 1,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epu32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 5,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epu32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 5,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epu32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 2,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epu32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpd128_mask ((__v4si) __X,
        (__v4si) __Y, 2,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epu64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 4,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epu64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 4,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epu64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 1,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epu64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 1,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epu64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 5,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epu64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 5,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epu64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 2,
        (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epu64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpq128_mask ((__v2di) __X,
        (__v2di) __Y, 2,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epi32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 4,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epi32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 4,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epi32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 1,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 1,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epi32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 5,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epi32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 5,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epi32_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 2,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epi32_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpd128_mask ((__v4si) __X,
       (__v4si) __Y, 2,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epi64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 4,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epi64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 4,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epi64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 1,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 1,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epi64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 5,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epi64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 5,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epi64_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 2,
       (__mmask8) __M);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epi64_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpq128_mask ((__v2di) __X,
       (__v2di) __Y, 2,
       (__mmask8) -1);
}
# 13643 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlintrin.h" 3 4
#pragma GCC pop_options
# 54 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512bwintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512bwintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512bw")
typedef short __v32hi __attribute__ ((__vector_size__ (64)));
typedef char __v64qi __attribute__ ((__vector_size__ (64)));
typedef unsigned long long __mmask64;
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktest_mask32_u8 (__mmask32 __A, __mmask32 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_ktestcsi (__A, __B);
  return (unsigned char) __builtin_ia32_ktestzsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktest_mask64_u8 (__mmask64 __A, __mmask64 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_ktestcdi (__A, __B);
  return (unsigned char) __builtin_ia32_ktestzdi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestz_mask32_u8 (__mmask32 __A, __mmask32 __B)
{
  return (unsigned char) __builtin_ia32_ktestzsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestz_mask64_u8 (__mmask64 __A, __mmask64 __B)
{
  return (unsigned char) __builtin_ia32_ktestzdi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestc_mask32_u8 (__mmask32 __A, __mmask32 __B)
{
  return (unsigned char) __builtin_ia32_ktestcsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestc_mask64_u8 (__mmask64 __A, __mmask64 __B)
{
  return (unsigned char) __builtin_ia32_ktestcdi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortest_mask32_u8 (__mmask32 __A, __mmask32 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_kortestcsi (__A, __B);
  return (unsigned char) __builtin_ia32_kortestzsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortest_mask64_u8 (__mmask64 __A, __mmask64 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_kortestcdi (__A, __B);
  return (unsigned char) __builtin_ia32_kortestzdi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestz_mask32_u8 (__mmask32 __A, __mmask32 __B)
{
  return (unsigned char) __builtin_ia32_kortestzsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestz_mask64_u8 (__mmask64 __A, __mmask64 __B)
{
  return (unsigned char) __builtin_ia32_kortestzdi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestc_mask32_u8 (__mmask32 __A, __mmask32 __B)
{
  return (unsigned char) __builtin_ia32_kortestcsi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestc_mask64_u8 (__mmask64 __A, __mmask64 __B)
{
  return (unsigned char) __builtin_ia32_kortestcdi (__A, __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kadd_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kaddsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kadd_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kadddi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtmask32_u32 (__mmask32 __A)
{
  return (unsigned int) __builtin_ia32_kmovd ((__mmask32) __A);
}
extern __inline unsigned long long
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtmask64_u64 (__mmask64 __A)
{
  return (unsigned long long) __builtin_ia32_kmovq ((__mmask64) __A);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtu32_mask32 (unsigned int __A)
{
  return (__mmask32) __builtin_ia32_kmovd ((__mmask32) __A);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtu64_mask64 (unsigned long long __A)
{
  return (__mmask64) __builtin_ia32_kmovq ((__mmask64) __A);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_load_mask32 (__mmask32 *__A)
{
  return (__mmask32) __builtin_ia32_kmovd (*__A);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_load_mask64 (__mmask64 *__A)
{
  return (__mmask64) __builtin_ia32_kmovq (*(__mmask64 *) __A);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_store_mask32 (__mmask32 *__A, __mmask32 __B)
{
  *(__mmask32 *) __A = __builtin_ia32_kmovd (__B);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_store_mask64 (__mmask64 *__A, __mmask64 __B)
{
  *(__mmask64 *) __A = __builtin_ia32_kmovq (__B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_knot_mask32 (__mmask32 __A)
{
  return (__mmask32) __builtin_ia32_knotsi ((__mmask32) __A);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_knot_mask64 (__mmask64 __A)
{
  return (__mmask64) __builtin_ia32_knotdi ((__mmask64) __A);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kor_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_korsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kor_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kordi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxnor_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kxnorsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxnor_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kxnordi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxor_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kxorsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxor_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kxordi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kand_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kandsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kand_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kanddi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kandn_mask32 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kandnsi ((__mmask32) __A, (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kandn_mask64 (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kandndi ((__mmask64) __A, (__mmask64) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_epi16 (__m512i __W, __mmask32 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdquhi512_mask ((__v32hi) __A,
          (__v32hi) __W,
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_epi16 (__mmask32 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdquhi512_mask ((__v32hi) __A,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_epi16 (__m512i __W, __mmask32 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddquhi512_mask ((const short *) __P,
           (__v32hi) __W,
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_epi16 (__mmask32 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddquhi512_mask ((const short *) __P,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_epi16 (void *__P, __mmask32 __U, __m512i __A)
{
  __builtin_ia32_storedquhi512_mask ((short *) __P,
         (__v32hi) __A,
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mov_epi8 (__m512i __W, __mmask64 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdquqi512_mask ((__v64qi) __A,
          (__v64qi) __W,
          (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mov_epi8 (__mmask64 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_movdquqi512_mask ((__v64qi) __A,
          (__v64qi)
          _mm512_setzero_si512 (),
          (__mmask64) __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kunpackw (__mmask32 __A, __mmask32 __B)
{
  return (__mmask32) __builtin_ia32_kunpcksi ((__mmask32) __A,
           (__mmask32) __B);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kunpackw_mask32 (__mmask16 __A, __mmask16 __B)
{
  return (__mmask32) __builtin_ia32_kunpcksi ((__mmask32) __A,
           (__mmask32) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_kunpackd (__mmask64 __A, __mmask64 __B)
{
  return (__mmask64) __builtin_ia32_kunpckdi ((__mmask64) __A,
           (__mmask64) __B);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kunpackd_mask64 (__mmask32 __A, __mmask32 __B)
{
  return (__mmask64) __builtin_ia32_kunpckdi ((__mmask64) __A,
           (__mmask64) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_loadu_epi8 (__m512i __W, __mmask64 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddquqi512_mask ((const char *) __P,
           (__v64qi) __W,
           (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_loadu_epi8 (__mmask64 __U, void const *__P)
{
  return (__m512i) __builtin_ia32_loaddquqi512_mask ((const char *) __P,
           (__v64qi)
           _mm512_setzero_si512 (),
           (__mmask64) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_storeu_epi8 (void *__P, __mmask64 __U, __m512i __A)
{
  __builtin_ia32_storedquqi512_mask ((char *) __P,
         (__v64qi) __A,
         (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sad_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psadbw512 ((__v64qi) __A,
          (__v64qi) __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi16_epi8 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovwb512_mask ((__v32hi) __A,
        (__v32qi) _mm256_undefined_si256(),
        (__mmask32) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi16_storeu_epi8 (void * __P, __mmask32 __M, __m512i __A)
{
  __builtin_ia32_pmovwb512mem_mask ((__v32qi *) __P, (__v32hi) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi16_epi8 (__m256i __O, __mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovwb512_mask ((__v32hi) __A,
        (__v32qi) __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi16_epi8 (__mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovwb512_mask ((__v32hi) __A,
        (__v32qi)
        _mm256_setzero_si256 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtsepi16_epi8 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovswb512_mask ((__v32hi) __A,
         (__v32qi)_mm256_undefined_si256(),
         (__mmask32) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi16_storeu_epi8 (void * __P, __mmask32 __M, __m512i __A)
{
  __builtin_ia32_pmovswb512mem_mask ((__v32qi *) __P, (__v32hi) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtsepi16_epi8 (__m256i __O, __mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovswb512_mask ((__v32hi) __A,
         (__v32qi)__O,
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtsepi16_epi8 (__mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovswb512_mask ((__v32hi) __A,
         (__v32qi)
         _mm256_setzero_si256 (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtusepi16_epi8 (__m512i __A)
{
  return (__m256i) __builtin_ia32_pmovuswb512_mask ((__v32hi) __A,
          (__v32qi)_mm256_undefined_si256(),
          (__mmask32) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi16_epi8 (__m256i __O, __mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovuswb512_mask ((__v32hi) __A,
          (__v32qi) __O,
          __M);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtusepi16_storeu_epi8 (void * __P, __mmask32 __M, __m512i __A)
{
  __builtin_ia32_pmovuswb512mem_mask ((__v32qi *) __P, (__v32hi) __A, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtusepi16_epi8 (__mmask32 __M, __m512i __A)
{
  return (__m256i) __builtin_ia32_pmovuswb512_mask ((__v32hi) __A,
          (__v32qi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastb_epi8 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastb512_mask ((__v16qi) __A,
             (__v64qi)_mm512_undefined_epi32(),
             (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastb_epi8 (__m512i __O, __mmask64 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastb512_mask ((__v16qi) __A,
             (__v64qi) __O,
             __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastb_epi8 (__mmask64 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastb512_mask ((__v16qi) __A,
             (__v64qi)
             _mm512_setzero_si512 (),
             __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_set1_epi8 (__m512i __O, __mmask64 __M, char __A)
{
  return (__m512i) __builtin_ia32_pbroadcastb512_gpr_mask (__A,
          (__v64qi) __O,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_set1_epi8 (__mmask64 __M, char __A)
{
  return (__m512i)
  __builtin_ia32_pbroadcastb512_gpr_mask (__A,
       (__v64qi)
       _mm512_setzero_si512 (),
       __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcastw_epi16 (__m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastw512_mask ((__v8hi) __A,
             (__v32hi)_mm512_undefined_epi32(),
             (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcastw_epi16 (__m512i __O, __mmask32 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastw512_mask ((__v8hi) __A,
             (__v32hi) __O,
             __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcastw_epi16 (__mmask32 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_pbroadcastw512_mask ((__v8hi) __A,
             (__v32hi)
             _mm512_setzero_si512 (),
             __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_set1_epi16 (__m512i __O, __mmask32 __M, short __A)
{
  return (__m512i) __builtin_ia32_pbroadcastw512_gpr_mask (__A,
          (__v32hi) __O,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_set1_epi16 (__mmask32 __M, short __A)
{
  return (__m512i)
  __builtin_ia32_pbroadcastw512_gpr_mask (__A,
       (__v32hi)
       _mm512_setzero_si512 (),
       __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mulhrs_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhrsw512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mulhrs_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
     __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhrsw512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v32hi) __W,
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mulhrs_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhrsw512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mulhi_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mulhi_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mulhi_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mulhi_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhuw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mulhi_epu16 (__m512i __W, __mmask32 __U, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhuw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi) __W,
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mulhi_epu16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmulhuw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mullo_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v32hu) __A * (__v32hu) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mullo_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_pmullw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mullo_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmullw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi8_epi16 (__m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbw512_mask ((__v32qi) __A,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi8_epi16 (__m512i __W, __mmask32 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbw512_mask ((__v32qi) __A,
          (__v32hi) __W,
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi8_epi16 (__mmask32 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovsxbw512_mask ((__v32qi) __A,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu8_epi16 (__m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbw512_mask ((__v32qi) __A,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu8_epi16 (__m512i __W, __mmask32 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbw512_mask ((__v32qi) __A,
          (__v32hi) __W,
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu8_epi16 (__mmask32 __U, __m256i __A)
{
  return (__m512i) __builtin_ia32_pmovzxbw512_mask ((__v32qi) __A,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarhi512_mask ((__v32hi) __B,
           (__v32hi) __A,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_epi16 (__mmask32 __M, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarhi512_mask ((__v32hi) __B,
           (__v32hi) __A,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_epi16 (__m512i __W, __mmask32 __M, __m512i __A,
          __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarhi512_mask ((__v32hi) __B,
           (__v32hi) __A,
           (__v32hi) __W,
           (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_epi16 (__m512i __A, __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varhi512_mask ((__v32hi) __I
                 ,
       (__v32hi) __A,
       (__v32hi) __B,
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_epi16 (__m512i __A, __mmask32 __U,
    __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varhi512_mask ((__v32hi) __I
                 ,
       (__v32hi) __A,
       (__v32hi) __B,
       (__mmask32)
       __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_epi16 (__m512i __A, __m512i __I,
     __mmask32 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermi2varhi512_mask ((__v32hi) __A,
       (__v32hi) __I
                 ,
       (__v32hi) __B,
       (__mmask32)
       __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_epi16 (__mmask32 __U, __m512i __A,
     __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varhi512_maskz ((__v32hi) __I
                  ,
        (__v32hi) __A,
        (__v32hi) __B,
        (__mmask32)
        __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_avg_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_avg_epu8 (__m512i __W, __mmask64 __U, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi) __W,
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_avg_epu8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v64qu) __A + (__v64qu) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_paddb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi) __W,
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v64qu) __A - (__v64qu) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_psubb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi) __W,
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_avg_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_avg_epu16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_avg_epu16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pavgw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_subs_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_subs_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_subs_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_subs_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi)
         _mm512_setzero_si512 (),
         (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_subs_epu8 (__m512i __W, __mmask64 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi) __W,
         (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_subs_epu8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi)
         _mm512_setzero_si512 (),
         (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_adds_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_adds_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_adds_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_adds_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi)
         _mm512_setzero_si512 (),
         (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_adds_epu8 (__m512i __W, __mmask64 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi) __W,
         (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_adds_epu8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusb512_mask ((__v64qi) __A,
         (__v64qi) __B,
         (__v64qi)
         _mm512_setzero_si512 (),
         (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sub_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v32hu) __A - (__v32hu) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sub_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_psubw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sub_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_subs_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_subs_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_subs_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_subs_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_subs_epu16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi) __W,
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_subs_epu16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psubusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_add_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v32hu) __A + (__v32hu) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_add_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_paddw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_add_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddw512_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_adds_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_adds_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_adds_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_adds_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_adds_epu16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi) __W,
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_adds_epu16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_paddusw512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v32hi)
         _mm512_setzero_si512 (),
         (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srl_epi16 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srl_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srl_epi16 (__mmask32 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psrlw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_packs_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packsswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi)
          _mm512_setzero_si512 (),
          (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sll_epi16 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psllw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sll_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m128i __B)
{
  return (__m512i) __builtin_ia32_psllw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sll_epi16 (__mmask32 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psllw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maddubs_epi16 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmaddubsw512_mask ((__v64qi) __X,
           (__v64qi) __Y,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_maddubs_epi16 (__m512i __W, __mmask32 __U, __m512i __X,
      __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmaddubsw512_mask ((__v64qi) __X,
           (__v64qi) __Y,
           (__v32hi) __W,
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_maddubs_epi16 (__mmask32 __U, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_pmaddubsw512_mask ((__v64qi) __X,
           (__v64qi) __Y,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_madd_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaddwd512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v16si)
         _mm512_setzero_si512 (),
         (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_madd_epi16 (__m512i __W, __mmask16 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaddwd512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v16si) __W,
         (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_madd_epi16 (__mmask16 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaddwd512_mask ((__v32hi) __A,
         (__v32hi) __B,
         (__v16si)
         _mm512_setzero_si512 (),
         (__mmask16) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi)
           _mm512_setzero_si512 (),
           (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
      __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi) __W,
           (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi)
           _mm512_setzero_si512 (),
           (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpackhi_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpackhi_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
       __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi) __W,
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpackhi_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpckhwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi)
           _mm512_setzero_si512 (),
           (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
      __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi) __W,
           (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklbw512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__v64qi)
           _mm512_setzero_si512 (),
           (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_unpacklo_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_unpacklo_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
       __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi) __W,
           (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_unpacklo_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_punpcklwd512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__v32hi)
           _mm512_setzero_si512 (),
           (__mmask32) __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epu8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __A,
          (__v64qi) __B, 0,
          (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epi8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_pcmpeqb512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epu8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __A,
          (__v64qi) __B, 0,
          __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epi8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_pcmpeqb512_mask ((__v64qi) __A,
           (__v64qi) __B,
           __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epu16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __A,
          (__v32hi) __B, 0,
          (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpeq_epi16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_pcmpeqw512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epu16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __A,
          (__v32hi) __B, 0,
          __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpeq_epi16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_pcmpeqw512_mask ((__v32hi) __A,
           (__v32hi) __B,
           __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epu8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __A,
          (__v64qi) __B, 6,
          (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epi8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_pcmpgtb512_mask ((__v64qi) __A,
           (__v64qi) __B,
           (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epu8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __A,
          (__v64qi) __B, 6,
          __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epi8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_pcmpgtb512_mask ((__v64qi) __A,
           (__v64qi) __B,
           __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epu16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __A,
          (__v32hi) __B, 6,
          (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpgt_epi16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_pcmpgtw512_mask ((__v32hi) __A,
           (__v32hi) __B,
           (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epu16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __A,
          (__v32hi) __B, 6,
          __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpgt_epi16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_pcmpgtw512_mask ((__v32hi) __A,
           (__v32hi) __B,
           __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movepi8_mask (__m512i __A)
{
  return (__mmask64) __builtin_ia32_cvtb2mask512 ((__v64qi) __A);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movepi16_mask (__m512i __A)
{
  return (__mmask32) __builtin_ia32_cvtw2mask512 ((__v32hi) __A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movm_epi8 (__mmask64 __A)
{
  return (__m512i) __builtin_ia32_cvtmask2b512 (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movm_epi16 (__mmask32 __A)
{
  return (__m512i) __builtin_ia32_cvtmask2w512 (__A);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_test_epi8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ptestmb512 ((__v64qi) __A,
      (__v64qi) __B,
      (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_test_epi8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ptestmb512 ((__v64qi) __A,
      (__v64qi) __B, __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_test_epi16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ptestmw512 ((__v32hi) __A,
      (__v32hi) __B,
      (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_test_epi16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ptestmw512 ((__v32hi) __A,
      (__v32hi) __B, __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_testn_epi8_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ptestnmb512 ((__v64qi) __A,
       (__v64qi) __B,
       (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_testn_epi8_mask (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_ptestnmb512 ((__v64qi) __A,
       (__v64qi) __B, __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_testn_epi16_mask (__m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ptestnmw512 ((__v32hi) __A,
       (__v32hi) __B,
       (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_testn_epi16_mask (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__mmask32) __builtin_ia32_ptestnmw512 ((__v32hi) __A,
       (__v32hi) __B, __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shuffle_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pshufb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shuffle_epi8 (__m512i __W, __mmask64 __U, __m512i __A,
     __m512i __B)
{
  return (__m512i) __builtin_ia32_pshufb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shuffle_epi8 (__mmask64 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pshufb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epu16 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epu16 (__m512i __W, __mmask32 __M, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_pminuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epi16 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epi16 (__m512i __W, __mmask32 __M, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epu8 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epu8 (__m512i __W, __mmask64 __M, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epi8 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epi8 (__m512i __W, __mmask64 __M, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epu8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epu8 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epu8 (__m512i __W, __mmask64 __M, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_pminub512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_min_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_min_epi8 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi)
        _mm512_setzero_si512 (),
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_min_epi8 (__m512i __W, __mmask64 __M, __m512i __A,
        __m512i __B)
{
  return (__m512i) __builtin_ia32_pminsb512_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__v64qi) __W,
        (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epi16 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epi16 (__m512i __W, __mmask32 __M, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxsw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_max_epu16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_max_epu16 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_max_epu16 (__m512i __W, __mmask32 __M, __m512i __A,
         __m512i __B)
{
  return (__m512i) __builtin_ia32_pmaxuw512_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sra_epi16 (__m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psraw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sra_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
         __m128i __B)
{
  return (__m512i) __builtin_ia32_psraw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sra_epi16 (__mmask32 __U, __m512i __A, __m128i __B)
{
  return (__m512i) __builtin_ia32_psraw512_mask ((__v32hi) __A,
       (__v8hi) __B,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srav_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psrav32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srav_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_psrav32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srav_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psrav32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_srlv_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psrlv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_srlv_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_psrlv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_srlv_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psrlv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_sllv_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psllv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_sllv_epi16 (__m512i __W, __mmask32 __U, __m512i __A,
   __m512i __B)
{
  return (__m512i) __builtin_ia32_psllv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi) __W,
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_sllv_epi16 (__mmask32 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_psllv32hi_mask ((__v32hi) __A,
        (__v32hi) __B,
        (__v32hi)
        _mm512_setzero_si512 (),
        (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_packs_epi16 (__m512i __W, __mmask64 __M, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_packsswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi) __W,
          (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_packs_epi16 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packsswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_packus_epi16 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packuswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi)
          _mm512_setzero_si512 (),
          (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_packus_epi16 (__m512i __W, __mmask64 __M, __m512i __A,
     __m512i __B)
{
  return (__m512i) __builtin_ia32_packuswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi) __W,
          (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_packus_epi16 (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packuswb512_mask ((__v32hi) __A,
          (__v32hi) __B,
          (__v64qi)
          _mm512_setzero_si512 (),
          (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_epi8 (__m512i __A)
{
  return (__m512i) __builtin_ia32_pabsb512_mask ((__v64qi) __A,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_epi8 (__m512i __W, __mmask64 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsb512_mask ((__v64qi) __A,
       (__v64qi) __W,
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_abs_epi8 (__mmask64 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsb512_mask ((__v64qi) __A,
       (__v64qi)
       _mm512_setzero_si512 (),
       (__mmask64) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_abs_epi16 (__m512i __A)
{
  return (__m512i) __builtin_ia32_pabsw512_mask ((__v32hi) __A,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_abs_epi16 (__m512i __W, __mmask32 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsw512_mask ((__v32hi) __A,
       (__v32hi) __W,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_abs_epi16 (__mmask32 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_pabsw512_mask ((__v32hi) __A,
       (__v32hi)
       _mm512_setzero_si512 (),
       (__mmask32) __U);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epu8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 4,
         (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epu8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 1,
         (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epu8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 5,
         (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epu8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 2,
         (__mmask64) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epu16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 4,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epu16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 1,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epu16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 5,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epu16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 2,
         (__mmask32) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epi8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 4,
        (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epi8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 1,
        (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epi8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 5,
        (__mmask64) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epi8_mask (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 2,
        (__mmask64) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpneq_epi16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 4,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmplt_epi16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 1,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmpge_epi16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 5,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cmple_epi16_mask (__mmask32 __M, __m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 2,
        (__mmask32) __M);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epu8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 4,
         (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epu8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 1,
         (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epu8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 5,
         (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epu8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_ucmpb512_mask ((__v64qi) __X,
         (__v64qi) __Y, 2,
         (__mmask64) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epu16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 4,
         (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epu16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 1,
         (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epu16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 5,
         (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epu16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpw512_mask ((__v32hi) __X,
         (__v32hi) __Y, 2,
         (__mmask32) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epi8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 4,
        (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epi8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 1,
        (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epi8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 5,
        (__mmask64) -1);
}
extern __inline __mmask64
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epi8_mask (__m512i __X, __m512i __Y)
{
  return (__mmask64) __builtin_ia32_cmpb512_mask ((__v64qi) __X,
        (__v64qi) __Y, 2,
        (__mmask64) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpneq_epi16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 4,
        (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmplt_epi16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 1,
        (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmpge_epi16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 5,
        (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cmple_epi16_mask (__m512i __X, __m512i __Y)
{
  return (__mmask32) __builtin_ia32_cmpw512_mask ((__v32hi) __X,
        (__v32hi) __Y, 2,
        (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_packs_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packssdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_packs_epi32 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packssdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_packs_epi32 (__m512i __W, __mmask32 __M, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_packssdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi) __W,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_packus_epi32 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packusdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          (__mmask32) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_packus_epi32 (__mmask32 __M, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_packusdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_packus_epi32 (__m512i __W, __mmask32 __M, __m512i __A,
     __m512i __B)
{
  return (__m512i) __builtin_ia32_packusdw512_mask ((__v16si) __A,
          (__v16si) __B,
          (__v32hi) __W,
          __M);
}
# 3298 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512bwintrin.h" 3 4
#pragma GCC pop_options
# 56 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512dqintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512dqintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512dq")
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktest_mask8_u8 (__mmask8 __A, __mmask8 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_ktestcqi (__A, __B);
  return (unsigned char) __builtin_ia32_ktestzqi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestz_mask8_u8 (__mmask8 __A, __mmask8 __B)
{
  return (unsigned char) __builtin_ia32_ktestzqi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestc_mask8_u8 (__mmask8 __A, __mmask8 __B)
{
  return (unsigned char) __builtin_ia32_ktestcqi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktest_mask16_u8 (__mmask16 __A, __mmask16 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_ktestchi (__A, __B);
  return (unsigned char) __builtin_ia32_ktestzhi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestz_mask16_u8 (__mmask16 __A, __mmask16 __B)
{
  return (unsigned char) __builtin_ia32_ktestzhi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_ktestc_mask16_u8 (__mmask16 __A, __mmask16 __B)
{
  return (unsigned char) __builtin_ia32_ktestchi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortest_mask8_u8 (__mmask8 __A, __mmask8 __B, unsigned char *__CF)
{
  *__CF = (unsigned char) __builtin_ia32_kortestcqi (__A, __B);
  return (unsigned char) __builtin_ia32_kortestzqi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestz_mask8_u8 (__mmask8 __A, __mmask8 __B)
{
  return (unsigned char) __builtin_ia32_kortestzqi (__A, __B);
}
extern __inline unsigned char
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kortestc_mask8_u8 (__mmask8 __A, __mmask8 __B)
{
  return (unsigned char) __builtin_ia32_kortestcqi (__A, __B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kadd_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_kaddqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kadd_mask16 (__mmask16 __A, __mmask16 __B)
{
  return (__mmask16) __builtin_ia32_kaddhi ((__mmask16) __A, (__mmask16) __B);
}
extern __inline unsigned int
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtmask8_u32 (__mmask8 __A)
{
  return (unsigned int) __builtin_ia32_kmovb ((__mmask8 ) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_cvtu32_mask8 (unsigned int __A)
{
  return (__mmask8) __builtin_ia32_kmovb ((__mmask8) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_load_mask8 (__mmask8 *__A)
{
  return (__mmask8) __builtin_ia32_kmovb (*(__mmask8 *) __A);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_store_mask8 (__mmask8 *__A, __mmask8 __B)
{
  *(__mmask8 *) __A = __builtin_ia32_kmovb (__B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_knot_mask8 (__mmask8 __A)
{
  return (__mmask8) __builtin_ia32_knotqi ((__mmask8) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kor_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_korqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxnor_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_kxnorqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kxor_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_kxorqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kand_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_kandqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_kandn_mask8 (__mmask8 __A, __mmask8 __B)
{
  return (__mmask8) __builtin_ia32_kandnqi ((__mmask8) __A, (__mmask8) __B);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_f64x2 (__m128d __A)
{
  return (__m512d)
  __builtin_ia32_broadcastf64x2_512_mask ((__v2df) __A,
       _mm512_undefined_pd (),
       (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_f64x2 (__m512d __O, __mmask8 __M, __m128d __A)
{
  return (__m512d) __builtin_ia32_broadcastf64x2_512_mask ((__v2df)
          __A,
          (__v8df)
          __O, __M);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_f64x2 (__mmask8 __M, __m128d __A)
{
  return (__m512d) __builtin_ia32_broadcastf64x2_512_mask ((__v2df)
          __A,
          (__v8df)
          _mm512_setzero_ps (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_i64x2 (__m128i __A)
{
  return (__m512i)
  __builtin_ia32_broadcasti64x2_512_mask ((__v2di) __A,
       _mm512_undefined_epi32 (),
       (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_i64x2 (__m512i __O, __mmask8 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti64x2_512_mask ((__v2di)
          __A,
          (__v8di)
          __O, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_i64x2 (__mmask8 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti64x2_512_mask ((__v2di)
          __A,
          (__v8di)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_f32x2 (__m128 __A)
{
  return (__m512)
  __builtin_ia32_broadcastf32x2_512_mask ((__v4sf) __A,
       (__v16sf)_mm512_undefined_ps (),
       (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_f32x2 (__m512 __O, __mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x2_512_mask ((__v4sf) __A,
         (__v16sf)
         __O, __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_f32x2 (__mmask16 __M, __m128 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x2_512_mask ((__v4sf) __A,
         (__v16sf)
         _mm512_setzero_ps (),
         __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_i32x2 (__m128i __A)
{
  return (__m512i)
  __builtin_ia32_broadcasti32x2_512_mask ((__v4si) __A,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_i32x2 (__m512i __O, __mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x2_512_mask ((__v4si)
          __A,
          (__v16si)
          __O, __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_i32x2 (__mmask16 __M, __m128i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x2_512_mask ((__v4si)
          __A,
          (__v16si)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_f32x8 (__m256 __A)
{
  return (__m512)
  __builtin_ia32_broadcastf32x8_512_mask ((__v8sf) __A,
       _mm512_undefined_ps (),
       (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_f32x8 (__m512 __O, __mmask16 __M, __m256 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x8_512_mask ((__v8sf) __A,
         (__v16sf)__O,
         __M);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_f32x8 (__mmask16 __M, __m256 __A)
{
  return (__m512) __builtin_ia32_broadcastf32x8_512_mask ((__v8sf) __A,
         (__v16sf)
         _mm512_setzero_ps (),
         __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_broadcast_i32x8 (__m256i __A)
{
  return (__m512i)
  __builtin_ia32_broadcasti32x8_512_mask ((__v8si) __A,
       (__v16si)
       _mm512_undefined_epi32 (),
       (__mmask16) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_broadcast_i32x8 (__m512i __O, __mmask16 __M, __m256i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x8_512_mask ((__v8si)
          __A,
          (__v16si)__O,
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_broadcast_i32x8 (__mmask16 __M, __m256i __A)
{
  return (__m512i) __builtin_ia32_broadcasti32x8_512_mask ((__v8si)
          __A,
          (__v16si)
          _mm512_setzero_si512 (),
          __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mullo_epi64 (__m512i __A, __m512i __B)
{
  return (__m512i) ((__v8du) __A * (__v8du) __B);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_mullo_epi64 (__m512i __W, __mmask8 __U, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_pmullq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di) __W,
        (__mmask8) __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_mullo_epi64 (__mmask8 __U, __m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_pmullq512_mask ((__v8di) __A,
        (__v8di) __B,
        (__v8di)
        _mm512_setzero_si512 (),
        (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_xor_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_xorpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_xor_pd (__m512d __W, __mmask8 __U, __m512d __A,
      __m512d __B)
{
  return (__m512d) __builtin_ia32_xorpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_xor_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_xorpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_xor_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_xorps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_xor_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_xorps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_xor_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_xorps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_or_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_orpd512_mask ((__v8df) __A,
      (__v8df) __B,
      (__v8df)
      _mm512_setzero_pd (),
      (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_or_pd (__m512d __W, __mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_orpd512_mask ((__v8df) __A,
      (__v8df) __B,
      (__v8df) __W,
      (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_or_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_orpd512_mask ((__v8df) __A,
      (__v8df) __B,
      (__v8df)
      _mm512_setzero_pd (),
      (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_or_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_orps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            (__v16sf)
            _mm512_setzero_ps (),
            (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_or_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_orps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            (__v16sf) __W,
            (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_or_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_orps512_mask ((__v16sf) __A,
            (__v16sf) __B,
            (__v16sf)
            _mm512_setzero_ps (),
            (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_and_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_andpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_and_pd (__m512d __W, __mmask8 __U, __m512d __A,
      __m512d __B)
{
  return (__m512d) __builtin_ia32_andpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df) __W,
       (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_and_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_andpd512_mask ((__v8df) __A,
       (__v8df) __B,
       (__v8df)
       _mm512_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_and_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_andps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_and_ps (__m512 __W, __mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_andps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf) __W,
      (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_and_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_andps512_mask ((__v16sf) __A,
      (__v16sf) __B,
      (__v16sf)
      _mm512_setzero_ps (),
      (__mmask16) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_andnot_pd (__m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_andnpd512_mask ((__v8df) __A,
        (__v8df) __B,
        (__v8df)
        _mm512_setzero_pd (),
        (__mmask8) -1);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_andnot_pd (__m512d __W, __mmask8 __U, __m512d __A,
         __m512d __B)
{
  return (__m512d) __builtin_ia32_andnpd512_mask ((__v8df) __A,
        (__v8df) __B,
        (__v8df) __W,
        (__mmask8) __U);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_andnot_pd (__mmask8 __U, __m512d __A, __m512d __B)
{
  return (__m512d) __builtin_ia32_andnpd512_mask ((__v8df) __A,
        (__v8df) __B,
        (__v8df)
        _mm512_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_andnot_ps (__m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_andnps512_mask ((__v16sf) __A,
       (__v16sf) __B,
       (__v16sf)
       _mm512_setzero_ps (),
       (__mmask16) -1);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_andnot_ps (__m512 __W, __mmask16 __U, __m512 __A,
         __m512 __B)
{
  return (__m512) __builtin_ia32_andnps512_mask ((__v16sf) __A,
       (__v16sf) __B,
       (__v16sf) __W,
       (__mmask16) __U);
}
extern __inline __m512
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_andnot_ps (__mmask16 __U, __m512 __A, __m512 __B)
{
  return (__m512) __builtin_ia32_andnps512_mask ((__v16sf) __A,
       (__v16sf) __B,
       (__v16sf)
       _mm512_setzero_ps (),
       (__mmask16) __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movepi32_mask (__m512i __A)
{
  return (__mmask16) __builtin_ia32_cvtd2mask512 ((__v16si) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movepi64_mask (__m512i __A)
{
  return (__mmask8) __builtin_ia32_cvtq2mask512 ((__v8di) __A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movm_epi32 (__mmask16 __A)
{
  return (__m512i) __builtin_ia32_cvtmask2d512 (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_movm_epi64 (__mmask8 __A)
{
  return (__m512i) __builtin_ia32_cvtmask2q512 (__A);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttpd_epi64 (__m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2qq512_mask ((__v8df) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttpd_epi64 (__m512i __W, __mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2qq512_mask ((__v8df) __A,
           (__v8di) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttpd_epi64 (__mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2qq512_mask ((__v8df) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttpd_epu64 (__m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2uqq512_mask ((__v8df) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) -1,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttpd_epu64 (__m512i __W, __mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2uqq512_mask ((__v8df) __A,
            (__v8di) __W,
            (__mmask8) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttpd_epu64 (__mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvttpd2uqq512_mask ((__v8df) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttps_epi64 (__m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2qq512_mask ((__v8sf) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttps_epi64 (__m512i __W, __mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2qq512_mask ((__v8sf) __A,
           (__v8di) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttps_epi64 (__mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2qq512_mask ((__v8sf) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvttps_epu64 (__m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2uqq512_mask ((__v8sf) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) -1,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvttps_epu64 (__m512i __W, __mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2uqq512_mask ((__v8sf) __A,
            (__v8di) __W,
            (__mmask8) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvttps_epu64 (__mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvttps2uqq512_mask ((__v8sf) __A,
            (__v8di)
            _mm512_setzero_si512 (),
            (__mmask8) __U,
            0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtpd_epi64 (__m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2qq512_mask ((__v8df) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtpd_epi64 (__m512i __W, __mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2qq512_mask ((__v8df) __A,
          (__v8di) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtpd_epi64 (__mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2qq512_mask ((__v8df) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtpd_epu64 (__m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2uqq512_mask ((__v8df) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtpd_epu64 (__m512i __W, __mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2uqq512_mask ((__v8df) __A,
           (__v8di) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtpd_epu64 (__mmask8 __U, __m512d __A)
{
  return (__m512i) __builtin_ia32_cvtpd2uqq512_mask ((__v8df) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtps_epi64 (__m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2qq512_mask ((__v8sf) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtps_epi64 (__m512i __W, __mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2qq512_mask ((__v8sf) __A,
          (__v8di) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtps_epi64 (__mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2qq512_mask ((__v8sf) __A,
          (__v8di)
          _mm512_setzero_si512 (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtps_epu64 (__m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2uqq512_mask ((__v8sf) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtps_epu64 (__m512i __W, __mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2uqq512_mask ((__v8sf) __A,
           (__v8di) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtps_epu64 (__mmask8 __U, __m256 __A)
{
  return (__m512i) __builtin_ia32_cvtps2uqq512_mask ((__v8sf) __A,
           (__v8di)
           _mm512_setzero_si512 (),
           (__mmask8) __U,
           0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi64_ps (__m512i __A)
{
  return (__m256) __builtin_ia32_cvtqq2ps512_mask ((__v8di) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) -1,
         0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_ps (__m256 __W, __mmask8 __U, __m512i __A)
{
  return (__m256) __builtin_ia32_cvtqq2ps512_mask ((__v8di) __A,
         (__v8sf) __W,
         (__mmask8) __U,
         0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi64_ps (__mmask8 __U, __m512i __A)
{
  return (__m256) __builtin_ia32_cvtqq2ps512_mask ((__v8di) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         (__mmask8) __U,
         0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu64_ps (__m512i __A)
{
  return (__m256) __builtin_ia32_cvtuqq2ps512_mask ((__v8di) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu64_ps (__m256 __W, __mmask8 __U, __m512i __A)
{
  return (__m256) __builtin_ia32_cvtuqq2ps512_mask ((__v8di) __A,
          (__v8sf) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu64_ps (__mmask8 __U, __m512i __A)
{
  return (__m256) __builtin_ia32_cvtuqq2ps512_mask ((__v8di) __A,
          (__v8sf)
          _mm256_setzero_ps (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepi64_pd (__m512i __A)
{
  return (__m512d) __builtin_ia32_cvtqq2pd512_mask ((__v8di) __A,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) -1,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepi64_pd (__m512d __W, __mmask8 __U, __m512i __A)
{
  return (__m512d) __builtin_ia32_cvtqq2pd512_mask ((__v8di) __A,
          (__v8df) __W,
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepi64_pd (__mmask8 __U, __m512i __A)
{
  return (__m512d) __builtin_ia32_cvtqq2pd512_mask ((__v8di) __A,
          (__v8df)
          _mm512_setzero_pd (),
          (__mmask8) __U,
          0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_cvtepu64_pd (__m512i __A)
{
  return (__m512d) __builtin_ia32_cvtuqq2pd512_mask ((__v8di) __A,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) -1,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_cvtepu64_pd (__m512d __W, __mmask8 __U, __m512i __A)
{
  return (__m512d) __builtin_ia32_cvtuqq2pd512_mask ((__v8di) __A,
           (__v8df) __W,
           (__mmask8) __U,
           0x04);
}
extern __inline __m512d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_cvtepu64_pd (__mmask8 __U, __m512i __A)
{
  return (__m512d) __builtin_ia32_cvtuqq2pd512_mask ((__v8di) __A,
           (__v8df)
           _mm512_setzero_pd (),
           (__mmask8) __U,
           0x04);
}
# 2676 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512dqintrin.h" 3 4
#pragma GCC pop_options
# 58 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlbwintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlbwintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vl,avx512bw")
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_epi8 (__m256i __W, __mmask32 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdquqi256_mask ((__v32qi) __A,
          (__v32qi) __W,
          (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_epi8 (__mmask32 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdquqi256_mask ((__v32qi) __A,
          (__v32qi)
          _mm256_setzero_si256 (),
          (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_epi8 (__m128i __W, __mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdquqi128_mask ((__v16qi) __A,
          (__v16qi) __W,
          (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_epi8 (__mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdquqi128_mask ((__v16qi) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_epi8 (void *__P, __mmask32 __U, __m256i __A)
{
  __builtin_ia32_storedquqi256_mask ((char *) __P,
         (__v32qi) __A,
         (__mmask32) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_epi8 (void *__P, __mmask16 __U, __m128i __A)
{
  __builtin_ia32_storedquqi128_mask ((char *) __P,
         (__v16qi) __A,
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_epi16 (__m256i __W, __mmask16 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddquhi256_mask ((const short *) __P,
           (__v16hi) __W,
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_epi16 (__mmask16 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddquhi256_mask ((const short *) __P,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_epi16 (__m128i __W, __mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddquhi128_mask ((const short *) __P,
           (__v8hi) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_epi16 (__mmask8 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddquhi128_mask ((const short *) __P,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mov_epi16 (__m256i __W, __mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdquhi256_mask ((__v16hi) __A,
          (__v16hi) __W,
          (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mov_epi16 (__mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_movdquhi256_mask ((__v16hi) __A,
          (__v16hi)
          _mm256_setzero_si256 (),
          (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mov_epi16 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdquhi128_mask ((__v8hi) __A,
          (__v8hi) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mov_epi16 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_movdquhi128_mask ((__v8hi) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_loadu_epi8 (__m256i __W, __mmask32 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddquqi256_mask ((const char *) __P,
           (__v32qi) __W,
           (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_loadu_epi8 (__mmask32 __U, void const *__P)
{
  return (__m256i) __builtin_ia32_loaddquqi256_mask ((const char *) __P,
           (__v32qi)
           _mm256_setzero_si256 (),
           (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_loadu_epi8 (__m128i __W, __mmask16 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddquqi128_mask ((const char *) __P,
           (__v16qi) __W,
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_loadu_epi8 (__mmask16 __U, void const *__P)
{
  return (__m128i) __builtin_ia32_loaddquqi128_mask ((const char *) __P,
           (__v16qi)
           _mm_setzero_si128 (),
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi16_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovwb256_mask ((__v16hi) __A,
        (__v16qi)_mm_undefined_si128(),
        (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi16_storeu_epi8 (void * __P, __mmask16 __M,__m256i __A)
{
  __builtin_ia32_pmovwb256mem_mask ((__v16qi *) __P , (__v16hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi16_epi8 (__m128i __O, __mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovwb256_mask ((__v16hi) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi16_epi8 (__mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovwb256_mask ((__v16hi) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtsepi16_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovswb128_mask ((__v8hi) __A,
         (__v16qi)_mm_undefined_si128(),
         (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi16_storeu_epi8 (void * __P, __mmask8 __M,__m128i __A)
{
  __builtin_ia32_pmovswb128mem_mask ((__v8qi *) __P , (__v8hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtsepi16_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovswb128_mask ((__v8hi) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtsepi16_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovswb128_mask ((__v8hi) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtsepi16_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovswb256_mask ((__v16hi) __A,
         (__v16qi)_mm_undefined_si128(),
         (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi16_storeu_epi8 (void * __P, __mmask16 __M,__m256i __A)
{
  __builtin_ia32_pmovswb256mem_mask ((__v16qi *) __P , (__v16hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtsepi16_epi8 (__m128i __O, __mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovswb256_mask ((__v16hi) __A,
         (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtsepi16_epi8 (__mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovswb256_mask ((__v16hi) __A,
         (__v16qi)
         _mm_setzero_si128 (),
         __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtusepi16_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb128_mask ((__v8hi) __A,
          (__v16qi)_mm_undefined_si128(),
          (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi16_storeu_epi8 (void * __P, __mmask8 __M,__m128i __A)
{
  __builtin_ia32_pmovuswb128mem_mask ((__v8qi *) __P , (__v8hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtusepi16_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb128_mask ((__v8hi) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtusepi16_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb128_mask ((__v8hi) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtusepi16_epi8 (__m256i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb256_mask ((__v16hi) __A,
          (__v16qi)_mm_undefined_si128(),
          (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi16_storeu_epi8 (void * __P, __mmask16 __M,__m256i __A)
{
  __builtin_ia32_pmovuswb256mem_mask ((__v16qi *) __P , (__v16hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtusepi16_epi8 (__m128i __O, __mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb256_mask ((__v16hi) __A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtusepi16_epi8 (__mmask16 __M, __m256i __A)
{
  return (__m128i) __builtin_ia32_pmovuswb256_mask ((__v16hi) __A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastb_epi8 (__m256i __O, __mmask32 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastb256_mask ((__v16qi) __A,
             (__v32qi) __O,
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastb_epi8 (__mmask32 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastb256_mask ((__v16qi) __A,
             (__v32qi)
             _mm256_setzero_si256 (),
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_set1_epi8 (__m256i __O, __mmask32 __M, char __A)
{
  return (__m256i) __builtin_ia32_pbroadcastb256_gpr_mask (__A,
          (__v32qi) __O,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_set1_epi8 (__mmask32 __M, char __A)
{
  return (__m256i) __builtin_ia32_pbroadcastb256_gpr_mask (__A,
          (__v32qi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcastb_epi8 (__m128i __O, __mmask16 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastb128_mask ((__v16qi) __A,
             (__v16qi) __O,
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcastb_epi8 (__mmask16 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastb128_mask ((__v16qi) __A,
             (__v16qi)
             _mm_setzero_si128 (),
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_set1_epi8 (__m128i __O, __mmask16 __M, char __A)
{
  return (__m128i) __builtin_ia32_pbroadcastb128_gpr_mask (__A,
          (__v16qi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_set1_epi8 (__mmask16 __M, char __A)
{
  return (__m128i) __builtin_ia32_pbroadcastb128_gpr_mask (__A,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcastw_epi16 (__m256i __O, __mmask16 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastw256_mask ((__v8hi) __A,
             (__v16hi) __O,
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcastw_epi16 (__mmask16 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_pbroadcastw256_mask ((__v8hi) __A,
             (__v16hi)
             _mm256_setzero_si256 (),
             __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_set1_epi16 (__m256i __O, __mmask16 __M, short __A)
{
  return (__m256i) __builtin_ia32_pbroadcastw256_gpr_mask (__A,
          (__v16hi) __O,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_set1_epi16 (__mmask16 __M, short __A)
{
  return (__m256i) __builtin_ia32_pbroadcastw256_gpr_mask (__A,
          (__v16hi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcastw_epi16 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastw128_mask ((__v8hi) __A,
             (__v8hi) __O,
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcastw_epi16 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pbroadcastw128_mask ((__v8hi) __A,
             (__v8hi)
             _mm_setzero_si128 (),
             __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_set1_epi16 (__m128i __O, __mmask8 __M, short __A)
{
  return (__m128i) __builtin_ia32_pbroadcastw128_gpr_mask (__A,
          (__v8hi) __O,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_set1_epi16 (__mmask8 __M, short __A)
{
  return (__m128i) __builtin_ia32_pbroadcastw128_gpr_mask (__A,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutexvar_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarhi256_mask ((__v16hi) __B,
           (__v16hi) __A,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_epi16 (__mmask16 __M, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarhi256_mask ((__v16hi) __B,
           (__v16hi) __A,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_epi16 (__m256i __W, __mmask16 __M, __m256i __A,
          __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarhi256_mask ((__v16hi) __B,
           (__v16hi) __A,
           (__v16hi) __W,
           (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutexvar_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarhi128_mask ((__v8hi) __B,
           (__v8hi) __A,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutexvar_epi16 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarhi128_mask ((__v8hi) __B,
           (__v8hi) __A,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutexvar_epi16 (__m128i __W, __mmask8 __M, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarhi128_mask ((__v8hi) __B,
           (__v8hi) __A,
           (__v8hi) __W,
           (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_epi16 (__m256i __A, __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varhi256_mask ((__v16hi) __I
                 ,
       (__v16hi) __A,
       (__v16hi) __B,
       (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_epi16 (__m256i __A, __mmask16 __U,
    __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varhi256_mask ((__v16hi) __I
                 ,
       (__v16hi) __A,
       (__v16hi) __B,
       (__mmask16)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_epi16 (__m256i __A, __m256i __I,
     __mmask16 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermi2varhi256_mask ((__v16hi) __A,
       (__v16hi) __I
                 ,
       (__v16hi) __B,
       (__mmask16)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_epi16 (__mmask16 __U, __m256i __A,
     __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varhi256_maskz ((__v16hi) __I
                  ,
        (__v16hi) __A,
        (__v16hi) __B,
        (__mmask16)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_epi16 (__m128i __A, __m128i __I, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varhi128_mask ((__v8hi) __I
                 ,
       (__v8hi) __A,
       (__v8hi) __B,
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_epi16 (__m128i __A, __mmask8 __U, __m128i __I,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varhi128_mask ((__v8hi) __I
                 ,
       (__v8hi) __A,
       (__v8hi) __B,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_epi16 (__m128i __A, __m128i __I, __mmask8 __U,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermi2varhi128_mask ((__v8hi) __A,
       (__v8hi) __I
                 ,
       (__v8hi) __B,
       (__mmask8)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_epi16 (__mmask8 __U, __m128i __A, __m128i __I,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varhi128_maskz ((__v8hi) __I
                  ,
        (__v8hi) __A,
        (__v8hi) __B,
        (__mmask8)
        __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_maddubs_epi16 (__m256i __W, __mmask16 __U, __m256i __X,
      __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmaddubsw256_mask ((__v32qi) __X,
           (__v32qi) __Y,
           (__v16hi) __W,
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_maddubs_epi16 (__mmask16 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmaddubsw256_mask ((__v32qi) __X,
           (__v32qi) __Y,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_maddubs_epi16 (__m128i __W, __mmask8 __U, __m128i __X,
   __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaddubsw128_mask ((__v16qi) __X,
           (__v16qi) __Y,
           (__v8hi) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_maddubs_epi16 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmaddubsw128_mask ((__v16qi) __X,
           (__v16qi) __Y,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_madd_epi16 (__m256i __W, __mmask8 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaddwd256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v8si) __W,
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_madd_epi16 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaddwd256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v8si)
         _mm256_setzero_si256 (),
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_madd_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaddwd128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v4si) __W,
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_madd_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaddwd128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v4si)
         _mm_setzero_si128 (),
         (__mmask8) __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movepi8_mask (__m128i __A)
{
  return (__mmask16) __builtin_ia32_cvtb2mask128 ((__v16qi) __A);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movepi8_mask (__m256i __A)
{
  return (__mmask32) __builtin_ia32_cvtb2mask256 ((__v32qi) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movepi16_mask (__m128i __A)
{
  return (__mmask8) __builtin_ia32_cvtw2mask128 ((__v8hi) __A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movepi16_mask (__m256i __A)
{
  return (__mmask16) __builtin_ia32_cvtw2mask256 ((__v16hi) __A);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movm_epi8 (__mmask16 __A)
{
  return (__m128i) __builtin_ia32_cvtmask2b128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movm_epi8 (__mmask32 __A)
{
  return (__m256i) __builtin_ia32_cvtmask2b256 (__A);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movm_epi16 (__mmask8 __A)
{
  return (__m128i) __builtin_ia32_cvtmask2w128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movm_epi16 (__mmask16 __A)
{
  return (__m256i) __builtin_ia32_cvtmask2w256 (__A);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_test_epi8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ptestmb128 ((__v16qi) __A,
      (__v16qi) __B,
      (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_test_epi8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ptestmb128 ((__v16qi) __A,
      (__v16qi) __B, __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_test_epi8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ptestmb256 ((__v32qi) __A,
      (__v32qi) __B,
      (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_test_epi8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ptestmb256 ((__v32qi) __A,
      (__v32qi) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_test_epi16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmw128 ((__v8hi) __A,
            (__v8hi) __B,
            (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_test_epi16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestmw128 ((__v8hi) __A,
            (__v8hi) __B, __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_test_epi16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ptestmw256 ((__v16hi) __A,
      (__v16hi) __B,
      (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_test_epi16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ptestmw256 ((__v16hi) __A,
      (__v16hi) __B, __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epu16 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminuw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epu16 (__m256i __W, __mmask16 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminuw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epu16 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminuw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epu16 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminuw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epi16 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epi16 (__m256i __W, __mmask16 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epu8 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxub256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epu8 (__m256i __W, __mmask32 __M, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxub256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epu8 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxub128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epu8 (__m128i __W, __mmask16 __M, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxub128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epi8 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epi8 (__m256i __W, __mmask32 __M, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epi8 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epi8 (__m128i __W, __mmask16 __M, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epu8 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminub256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epu8 (__m256i __W, __mmask32 __M, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pminub256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epu8 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminub128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epu8 (__m128i __W, __mmask16 __M, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_pminub128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_min_epi8 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_min_epi8 (__m256i __W, __mmask32 __M, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pminsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epi8 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epi8 (__m128i __W, __mmask16 __M, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epi16 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epi16 (__m256i __W, __mmask16 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epi16 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epi16 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_max_epu16 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxuw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_max_epu16 (__m256i __W, __mmask16 __M, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pmaxuw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_max_epu16 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxuw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_max_epu16 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pmaxuw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_min_epi16 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_min_epi16 (__m128i __W, __mmask8 __M, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pminsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __M);
}
# 2039 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vlbwintrin.h" 3 4
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epi8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 4,
        (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epi8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 1,
        (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epi8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 5,
        (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epi8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 2,
        (__mmask32) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epi16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 4,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epi16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 1,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epi16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 5,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epi16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 2,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epu8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 4,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epu8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 1,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epu8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 5,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epu8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 2,
         (__mmask16) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epu16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 4,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epu16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 1,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epu16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 5,
        (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epu16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 2,
        (__mmask8) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epi8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 4,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 1,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epi8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 5,
        (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epi8_mask (__m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 2,
        (__mmask16) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpneq_epi16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 4,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmplt_epi16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 1,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpge_epi16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 5,
       (__mmask8) -1);
}
extern __inline __mmask8
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmple_epi16_mask (__m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 2,
       (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mulhrs_epi16 (__m256i __W, __mmask16 __U, __m256i __X,
     __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmulhrsw256_mask ((__v16hi) __X,
          (__v16hi) __Y,
          (__v16hi) __W,
          (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mulhrs_epi16 (__mmask16 __U, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_pmulhrsw256_mask ((__v16hi) __X,
          (__v16hi) __Y,
          (__v16hi)
          _mm256_setzero_si256 (),
          (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mulhi_epu16 (__m256i __W, __mmask16 __U, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulhuw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi) __W,
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mulhi_epu16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulhuw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi)
         _mm256_setzero_si256 (),
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mulhi_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulhw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mulhi_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmulhw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mulhi_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulhw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mulhi_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulhw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mulhi_epu16 (__m128i __W, __mmask8 __U, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulhuw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi) __W,
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mulhi_epu16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmulhuw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi)
         _mm_setzero_si128 (),
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mulhrs_epi16 (__m128i __W, __mmask8 __U, __m128i __X,
         __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmulhrsw128_mask ((__v8hi) __X,
          (__v8hi) __Y,
          (__v8hi) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mulhrs_epi16 (__mmask8 __U, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_pmulhrsw128_mask ((__v8hi) __X,
          (__v8hi) __Y,
          (__v8hi)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mullo_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_pmullw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mullo_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmullw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mullo_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_pmullw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mullo_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmullw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi8_epi16 (__m256i __W, __mmask16 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbw256_mask ((__v16qi) __A,
          (__v16hi) __W,
          (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi8_epi16 (__mmask16 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovsxbw256_mask ((__v16qi) __A,
          (__v16hi)
          _mm256_setzero_si256 (),
          (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi8_epi16 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbw128_mask ((__v16qi) __A,
          (__v8hi) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi8_epi16 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovsxbw128_mask ((__v16qi) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu8_epi16 (__m256i __W, __mmask16 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbw256_mask ((__v16qi) __A,
          (__v16hi) __W,
          (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu8_epi16 (__mmask16 __U, __m128i __A)
{
  return (__m256i) __builtin_ia32_pmovzxbw256_mask ((__v16qi) __A,
          (__v16hi)
          _mm256_setzero_si256 (),
          (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu8_epi16 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbw128_mask ((__v16qi) __A,
          (__v8hi) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu8_epi16 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovzxbw128_mask ((__v16qi) __A,
          (__v8hi)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_avg_epu8 (__m256i __W, __mmask32 __U, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_pavgb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi) __W,
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_avg_epu8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pavgb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi)
       _mm256_setzero_si256 (),
       (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_avg_epu8 (__m128i __W, __mmask16 __U, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_pavgb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi) __W,
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_avg_epu8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pavgb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi)
       _mm_setzero_si128 (),
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_avg_epu16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_pavgw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_avg_epu16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pavgw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_avg_epu16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_pavgw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_avg_epu16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pavgw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_paddb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi) __W,
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi)
       _mm256_setzero_si256 (),
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_add_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_paddw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_add_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_adds_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_paddsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_adds_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_adds_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_paddsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_adds_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_adds_epu8 (__m256i __W, __mmask32 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_paddusb256_mask ((__v32qi) __A,
         (__v32qi) __B,
         (__v32qi) __W,
         (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_adds_epu8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddusb256_mask ((__v32qi) __A,
         (__v32qi) __B,
         (__v32qi)
         _mm256_setzero_si256 (),
         (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_adds_epu16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_paddusw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi) __W,
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_adds_epu16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_paddusw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi)
         _mm256_setzero_si256 (),
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
        __m256i __B)
{
  return (__m256i) __builtin_ia32_psubb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi) __W,
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__v32qi)
       _mm256_setzero_si256 (),
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sub_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_psubw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sub_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubw256_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_subs_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_psubsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_subs_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubsb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_subs_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_psubsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_subs_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubsw256_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_subs_epu8 (__m256i __W, __mmask32 __U, __m256i __A,
         __m256i __B)
{
  return (__m256i) __builtin_ia32_psubusb256_mask ((__v32qi) __A,
         (__v32qi) __B,
         (__v32qi) __W,
         (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_subs_epu8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubusb256_mask ((__v32qi) __A,
         (__v32qi) __B,
         (__v32qi)
         _mm256_setzero_si256 (),
         (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_subs_epu16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_psubusw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi) __W,
         (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_subs_epu16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psubusw256_mask ((__v16hi) __A,
         (__v16hi) __B,
         (__v16hi)
         _mm256_setzero_si256 (),
         (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_paddb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi) __W,
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi)
       _mm_setzero_si128 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_add_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_paddw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_add_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
      __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhbw256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__v32qi) __W,
           (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhbw256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__v32qi)
           _mm256_setzero_si256 (),
           (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
   __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhbw128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__v16qi) __W,
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhbw128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__v16qi)
           _mm_setzero_si128 (),
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpackhi_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhwd256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__v16hi) __W,
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpackhi_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpckhwd256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpackhi_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhwd128_mask ((__v8hi) __A,
           (__v8hi) __B,
           (__v8hi) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpackhi_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpckhwd128_mask ((__v8hi) __A,
           (__v8hi) __B,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
      __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklbw256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__v32qi) __W,
           (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklbw256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__v32qi)
           _mm256_setzero_si256 (),
           (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
   __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklbw128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__v16qi) __W,
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklbw128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__v16qi)
           _mm_setzero_si128 (),
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_unpacklo_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
       __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklwd256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__v16hi) __W,
           (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_unpacklo_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_punpcklwd256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__v16hi)
           _mm256_setzero_si256 (),
           (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_unpacklo_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
    __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklwd128_mask ((__v8hi) __A,
           (__v8hi) __B,
           (__v8hi) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_unpacklo_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_punpcklwd128_mask ((__v8hi) __A,
           (__v8hi) __B,
           (__v8hi)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqb128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epu8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __A,
          (__v16qi) __B, 0,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epu8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __A,
          (__v16qi) __B, 0,
          __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epi8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqb128_mask ((__v16qi) __A,
           (__v16qi) __B,
           __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epu8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __A,
          (__v32qi) __B, 0,
          (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_pcmpeqb256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epu8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __A,
          (__v32qi) __B, 0,
          __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epi8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_pcmpeqb256_mask ((__v32qi) __A,
           (__v32qi) __B,
           __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epu16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __A,
         (__v8hi) __B, 0,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpeq_epi16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqw128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epu16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __A,
         (__v8hi) __B, 0, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpeq_epi16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpeqw128_mask ((__v8hi) __A,
          (__v8hi) __B, __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epu16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __A,
          (__v16hi) __B, 0,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpeq_epi16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqw256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epu16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __A,
          (__v16hi) __B, 0,
          __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpeq_epi16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_pcmpeqw256_mask ((__v16hi) __A,
           (__v16hi) __B,
           __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epu8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __A,
          (__v16qi) __B, 6,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtb128_mask ((__v16qi) __A,
           (__v16qi) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epu8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __A,
          (__v16qi) __B, 6,
          __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epi8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtb128_mask ((__v16qi) __A,
           (__v16qi) __B,
           __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epu8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __A,
          (__v32qi) __B, 6,
          (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_pcmpgtb256_mask ((__v32qi) __A,
           (__v32qi) __B,
           (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epu8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __A,
          (__v32qi) __B, 6,
          __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epi8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_pcmpgtb256_mask ((__v32qi) __A,
           (__v32qi) __B,
           __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epu16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __A,
         (__v8hi) __B, 6,
         (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cmpgt_epi16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtw128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epu16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __A,
         (__v8hi) __B, 6, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpgt_epi16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_pcmpgtw128_mask ((__v8hi) __A,
          (__v8hi) __B, __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epu16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __A,
          (__v16hi) __B, 6,
          (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpgt_epi16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtw256_mask ((__v16hi) __A,
           (__v16hi) __B,
           (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epu16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __A,
          (__v16hi) __B, 6,
          __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpgt_epi16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_pcmpgtw256_mask ((__v16hi) __A,
           (__v16hi) __B,
           __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_testn_epi8_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmb128 ((__v16qi) __A,
       (__v16qi) __B,
       (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_testn_epi8_mask (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmb128 ((__v16qi) __A,
       (__v16qi) __B, __U);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testn_epi8_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ptestnmb256 ((__v32qi) __A,
       (__v32qi) __B,
       (__mmask32) -1);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_testn_epi8_mask (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_ptestnmb256 ((__v32qi) __A,
       (__v32qi) __B, __U);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_testn_epi16_mask (__m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmw128 ((__v8hi) __A,
      (__v8hi) __B,
      (__mmask8) -1);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_testn_epi16_mask (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__mmask8) __builtin_ia32_ptestnmw128 ((__v8hi) __A,
      (__v8hi) __B, __U);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_testn_epi16_mask (__m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmw256 ((__v16hi) __A,
       (__v16hi) __B,
       (__mmask16) -1);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_testn_epi16_mask (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__mmask16) __builtin_ia32_ptestnmw256 ((__v16hi) __A,
       (__v16hi) __B, __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shuffle_epi8 (__m256i __W, __mmask32 __U, __m256i __A,
     __m256i __B)
{
  return (__m256i) __builtin_ia32_pshufb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi) __W,
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shuffle_epi8 (__mmask32 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pshufb256_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__v32qi)
        _mm256_setzero_si256 (),
        (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shuffle_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_pshufb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shuffle_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pshufb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_packs_epi16 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_packsswb256_mask ((__v16hi) __A,
          (__v16hi) __B,
          (__v32qi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_packs_epi16 (__m256i __W, __mmask32 __M, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_packsswb256_mask ((__v16hi) __A,
          (__v16hi) __B,
          (__v32qi) __W,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_packs_epi16 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_packsswb128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_packs_epi16 (__m128i __W, __mmask16 __M, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_packsswb128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__v16qi) __W,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_packus_epi16 (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_packuswb256_mask ((__v16hi) __A,
          (__v16hi) __B,
          (__v32qi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_packus_epi16 (__m256i __W, __mmask32 __M, __m256i __A,
     __m256i __B)
{
  return (__m256i) __builtin_ia32_packuswb256_mask ((__v16hi) __A,
          (__v16hi) __B,
          (__v32qi) __W,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_packus_epi16 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_packuswb128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__v16qi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_packus_epi16 (__m128i __W, __mmask16 __M, __m128i __A,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_packuswb128_mask ((__v8hi) __A,
          (__v8hi) __B,
          (__v16qi) __W,
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_abs_epi8 (__m256i __W, __mmask32 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsb256_mask ((__v32qi) __A,
       (__v32qi) __W,
       (__mmask32) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_abs_epi8 (__mmask32 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsb256_mask ((__v32qi) __A,
       (__v32qi)
       _mm256_setzero_si256 (),
       (__mmask32) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_abs_epi8 (__m128i __W, __mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsb128_mask ((__v16qi) __A,
       (__v16qi) __W,
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_abs_epi8 (__mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsb128_mask ((__v16qi) __A,
       (__v16qi)
       _mm_setzero_si128 (),
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_abs_epi16 (__m256i __W, __mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsw256_mask ((__v16hi) __A,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_abs_epi16 (__mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_pabsw256_mask ((__v16hi) __A,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_abs_epi16 (__m128i __W, __mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsw128_mask ((__v8hi) __A,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_abs_epi16 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_pabsw128_mask ((__v8hi) __A,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epu8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 4,
         (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epu8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 1,
         (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epu8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 5,
         (__mmask32) -1);
}
extern __inline __mmask32
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epu8_mask (__m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 2,
         (__mmask32) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpneq_epu16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 4,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmplt_epu16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 1,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmpge_epu16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 5,
         (__mmask16) -1);
}
extern __inline __mmask16
  __attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cmple_epu16_mask (__m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 2,
         (__mmask16) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_storeu_epi16 (void *__P, __mmask16 __U, __m256i __A)
{
  __builtin_ia32_storedquhi256_mask ((short *) __P,
         (__v16hi) __A,
         (__mmask16) __U);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_storeu_epi16 (void *__P, __mmask8 __U, __m128i __A)
{
  __builtin_ia32_storedquhi128_mask ((short *) __P,
         (__v8hi) __A,
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_adds_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_paddsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_subs_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psubsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_subs_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_subs_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_psubsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_subs_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_subs_epu8 (__m128i __W, __mmask16 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psubusb128_mask ((__v16qi) __A,
         (__v16qi) __B,
         (__v16qi) __W,
         (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_subs_epu8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubusb128_mask ((__v16qi) __A,
         (__v16qi) __B,
         (__v16qi)
         _mm_setzero_si128 (),
         (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_subs_epu16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_psubusw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi) __W,
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_subs_epu16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubusw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi)
         _mm_setzero_si128 (),
         (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srl_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psrlw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srl_epi16 (__mmask16 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psrlw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srl_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srl_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sra_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psraw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sra_epi16 (__mmask16 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psraw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sra_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psraw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sra_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psraw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_adds_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddsw128_mask ((__v8hi) __A,
        (__v8hi) __B,
        (__v8hi)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_adds_epu8 (__m128i __W, __mmask16 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_paddusb128_mask ((__v16qi) __A,
         (__v16qi) __B,
         (__v16qi) __W,
         (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_adds_epu8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddusb128_mask ((__v16qi) __A,
         (__v16qi) __B,
         (__v16qi)
         _mm_setzero_si128 (),
         (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_adds_epu16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_paddusw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi) __W,
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_adds_epu16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddusw128_mask ((__v8hi) __A,
         (__v8hi) __B,
         (__v8hi)
         _mm_setzero_si128 (),
         (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
     __m128i __B)
{
  return (__m128i) __builtin_ia32_psubb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi) __W,
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__v16qi)
       _mm_setzero_si128 (),
       (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sub_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psubw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sub_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psubw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_adds_epi8 (__m128i __W, __mmask16 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_paddsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi) __W,
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_adds_epi8 (__mmask16 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_paddsb128_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi16_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_pmovwb128_mask ((__v8hi) __A,
        (__v16qi)_mm_undefined_si128(),
        (__mmask8) -1);
}
extern __inline void
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi16_storeu_epi8 (void * __P, __mmask8 __M,__m128i __A)
{
  __builtin_ia32_pmovwb128mem_mask ((__v8qi *) __P , (__v8hi) __A, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi16_epi8 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovwb128_mask ((__v8hi) __A,
        (__v16qi) __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi16_epi8 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_pmovwb128_mask ((__v8hi) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srav_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psrav16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srav_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_psrav16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srav_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psrav16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srav_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrav8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srav_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_psrav8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srav_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrav8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_srlv_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psrlv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_srlv_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_psrlv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_srlv_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psrlv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_srlv_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_srlv_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_srlv_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psrlv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_sllv_epi16 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psllv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sllv_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
   __m256i __B)
{
  return (__m256i) __builtin_ia32_psllv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi) __W,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sllv_epi16 (__mmask16 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_psllv16hi_mask ((__v16hi) __A,
        (__v16hi) __B,
        (__v16hi)
        _mm256_setzero_si256 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sllv_epi16 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psllv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sllv_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_psllv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sllv_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psllv8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_sll_epi16 (__m128i __W, __mmask8 __U, __m128i __A,
      __m128i __B)
{
  return (__m128i) __builtin_ia32_psllw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi) __W,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_sll_epi16 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_psllw128_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_sll_epi16 (__m256i __W, __mmask16 __U, __m256i __A,
         __m128i __B)
{
  return (__m256i) __builtin_ia32_psllw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi) __W,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_sll_epi16 (__mmask16 __U, __m256i __A, __m128i __B)
{
  return (__m256i) __builtin_ia32_psllw256_mask ((__v16hi) __A,
       (__v8hi) __B,
       (__v16hi)
       _mm256_setzero_si256 (),
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_packus_epi32 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_packusdw256_mask ((__v8si) __A,
          (__v8si) __B,
          (__v16hi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_packus_epi32 (__m256i __W, __mmask16 __M, __m256i __A,
     __m256i __B)
{
  return (__m256i) __builtin_ia32_packusdw256_mask ((__v8si) __A,
          (__v8si) __B,
          (__v16hi) __W,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_packus_epi32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_packusdw128_mask ((__v4si) __A,
          (__v4si) __B,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_packus_epi32 (__m128i __W, __mmask16 __M, __m128i __A,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_packusdw128_mask ((__v4si) __A,
          (__v4si) __B,
          (__v8hi) __W, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_packs_epi32 (__mmask16 __M, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_packssdw256_mask ((__v8si) __A,
          (__v8si) __B,
          (__v16hi)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_packs_epi32 (__m256i __W, __mmask16 __M, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_packssdw256_mask ((__v8si) __A,
          (__v8si) __B,
          (__v16hi) __W,
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_packs_epi32 (__mmask8 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_packssdw128_mask ((__v4si) __A,
          (__v4si) __B,
          (__v8hi)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_packs_epi32 (__m128i __W, __mmask16 __M, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_packssdw128_mask ((__v4si) __A,
          (__v4si) __B,
          (__v8hi) __W, __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epu8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 4,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epu8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 1,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epu8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 5,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epu8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpb128_mask ((__v16qi) __X,
         (__v16qi) __Y, 2,
         (__mmask16) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epu16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 4,
        (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epu16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 1,
        (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epu16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 5,
        (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epu16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_ucmpw128_mask ((__v8hi) __X,
        (__v8hi) __Y, 2,
        (__mmask8) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epi8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 4,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epi8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 1,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epi8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 5,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epi8_mask (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__mmask16) __builtin_ia32_cmpb128_mask ((__v16qi) __X,
        (__v16qi) __Y, 2,
        (__mmask16) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpneq_epi16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 4,
       (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmplt_epi16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 1,
       (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmpge_epi16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 5,
       (__mmask8) __M);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cmple_epi16_mask (__mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__mmask8) __builtin_ia32_cmpw128_mask ((__v8hi) __X,
       (__v8hi) __Y, 2,
       (__mmask8) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epu8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 4,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epu8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 1,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epu8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 5,
         (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epu8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_ucmpb256_mask ((__v32qi) __X,
         (__v32qi) __Y, 2,
         (__mmask32) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epu16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 4,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epu16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 1,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epu16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 5,
         (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epu16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_ucmpw256_mask ((__v16hi) __X,
         (__v16hi) __Y, 2,
         (__mmask16) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epi8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 4,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epi8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 1,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epi8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 5,
        (__mmask32) __M);
}
extern __inline __mmask32
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epi8_mask (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__mmask32) __builtin_ia32_cmpb256_mask ((__v32qi) __X,
        (__v32qi) __Y, 2,
        (__mmask32) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpneq_epi16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 4,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmplt_epi16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 1,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmpge_epi16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 5,
        (__mmask16) __M);
}
extern __inline __mmask16
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cmple_epi16_mask (__mmask16 __M, __m256i __X, __m256i __Y)
{
  return (__mmask16) __builtin_ia32_cmpw256_mask ((__v16hi) __X,
        (__v16hi) __Y, 2,
        (__mmask16) __M);
}
#pragma GCC pop_options
# 60 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vldqintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vldqintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vl,avx512dq")
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttpd_epi64 (__m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2qq256_mask ((__v4df) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttpd_epi64 (__m256i __W, __mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2qq256_mask ((__v4df) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttpd_epi64 (__mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2qq256_mask ((__v4df) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttpd_epi64 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2qq128_mask ((__v2df) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttpd_epi64 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2qq128_mask ((__v2df) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttpd_epi64 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2qq128_mask ((__v2df) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttpd_epu64 (__m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2uqq256_mask ((__v4df) __A,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttpd_epu64 (__m256i __W, __mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2uqq256_mask ((__v4df) __A,
            (__v4di) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttpd_epu64 (__mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvttpd2uqq256_mask ((__v4df) __A,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttpd_epu64 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2uqq128_mask ((__v2df) __A,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttpd_epu64 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2uqq128_mask ((__v2df) __A,
            (__v2di) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttpd_epu64 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvttpd2uqq128_mask ((__v2df) __A,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtpd_epi64 (__m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2qq256_mask ((__v4df) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtpd_epi64 (__m256i __W, __mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2qq256_mask ((__v4df) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtpd_epi64 (__mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2qq256_mask ((__v4df) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_epi64 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2qq128_mask ((__v2df) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtpd_epi64 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2qq128_mask ((__v2df) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtpd_epi64 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2qq128_mask ((__v2df) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtpd_epu64 (__m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2uqq256_mask ((__v4df) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtpd_epu64 (__m256i __W, __mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2uqq256_mask ((__v4df) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtpd_epu64 (__mmask8 __U, __m256d __A)
{
  return (__m256i) __builtin_ia32_cvtpd2uqq256_mask ((__v4df) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtpd_epu64 (__m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2uqq128_mask ((__v2df) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtpd_epu64 (__m128i __W, __mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2uqq128_mask ((__v2df) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtpd_epu64 (__mmask8 __U, __m128d __A)
{
  return (__m128i) __builtin_ia32_cvtpd2uqq128_mask ((__v2df) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttps_epi64 (__m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2qq256_mask ((__v4sf) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttps_epi64 (__m256i __W, __mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2qq256_mask ((__v4sf) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttps_epi64 (__mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2qq256_mask ((__v4sf) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttps_epi64 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2qq128_mask ((__v4sf) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttps_epi64 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2qq128_mask ((__v4sf) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttps_epi64 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2qq128_mask ((__v4sf) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvttps_epu64 (__m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2uqq256_mask ((__v4sf) __A,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvttps_epu64 (__m256i __W, __mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2uqq256_mask ((__v4sf) __A,
            (__v4di) __W,
            (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvttps_epu64 (__mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvttps2uqq256_mask ((__v4sf) __A,
            (__v4di)
            _mm256_setzero_si256 (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvttps_epu64 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2uqq128_mask ((__v4sf) __A,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvttps_epu64 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2uqq128_mask ((__v4sf) __A,
            (__v2di) __W,
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvttps_epu64 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvttps2uqq128_mask ((__v4sf) __A,
            (__v2di)
            _mm_setzero_si128 (),
            (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_f64x2 (__m128d __A)
{
  return (__m256d) __builtin_ia32_broadcastf64x2_256_mask ((__v2df)
          __A,
                 (__v4df)_mm256_undefined_pd(),
          (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_f64x2 (__m256d __O, __mmask8 __M, __m128d __A)
{
  return (__m256d) __builtin_ia32_broadcastf64x2_256_mask ((__v2df)
          __A,
          (__v4df)
          __O, __M);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_f64x2 (__mmask8 __M, __m128d __A)
{
  return (__m256d) __builtin_ia32_broadcastf64x2_256_mask ((__v2df)
          __A,
          (__v4df)
          _mm256_setzero_ps (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_i64x2 (__m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti64x2_256_mask ((__v2di)
          __A,
                 (__v4di)_mm256_undefined_si256(),
          (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_i64x2 (__m256i __O, __mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti64x2_256_mask ((__v2di)
          __A,
          (__v4di)
          __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_i64x2 (__mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti64x2_256_mask ((__v2di)
          __A,
          (__v4di)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_f32x2 (__m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x2_256_mask ((__v4sf) __A,
                (__v8sf)_mm256_undefined_ps(),
         (__mmask8) -1);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_f32x2 (__m256 __O, __mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x2_256_mask ((__v4sf) __A,
         (__v8sf) __O,
         __M);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_f32x2 (__mmask8 __M, __m128 __A)
{
  return (__m256) __builtin_ia32_broadcastf32x2_256_mask ((__v4sf) __A,
         (__v8sf)
         _mm256_setzero_ps (),
         __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_broadcast_i32x2 (__m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x2_256_mask ((__v4si)
          __A,
                (__v8si)_mm256_undefined_si256(),
          (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_broadcast_i32x2 (__m256i __O, __mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x2_256_mask ((__v4si)
          __A,
          (__v8si)
          __O, __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_broadcast_i32x2 (__mmask8 __M, __m128i __A)
{
  return (__m256i) __builtin_ia32_broadcasti32x2_256_mask ((__v4si)
          __A,
          (__v8si)
          _mm256_setzero_si256 (),
          __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_broadcast_i32x2 (__m128i __A)
{
  return (__m128i) __builtin_ia32_broadcasti32x2_128_mask ((__v4si)
          __A,
                (__v4si)_mm_undefined_si128(),
          (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_broadcast_i32x2 (__m128i __O, __mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_broadcasti32x2_128_mask ((__v4si)
          __A,
          (__v4si)
          __O, __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_broadcast_i32x2 (__mmask8 __M, __m128i __A)
{
  return (__m128i) __builtin_ia32_broadcasti32x2_128_mask ((__v4si)
          __A,
          (__v4si)
          _mm_setzero_si128 (),
          __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mullo_epi64 (__m256i __A, __m256i __B)
{
  return (__m256i) ((__v4du) __A * (__v4du) __B);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_mullo_epi64 (__m256i __W, __mmask8 __U, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_pmullq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di) __W,
        (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_mullo_epi64 (__mmask8 __U, __m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_pmullq256_mask ((__v4di) __A,
        (__v4di) __B,
        (__v4di)
        _mm256_setzero_si256 (),
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mullo_epi64 (__m128i __A, __m128i __B)
{
  return (__m128i) ((__v2du) __A * (__v2du) __B);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_mullo_epi64 (__m128i __W, __mmask8 __U, __m128i __A,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_pmullq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di) __W,
        (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_mullo_epi64 (__mmask8 __U, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_pmullq128_mask ((__v2di) __A,
        (__v2di) __B,
        (__v2di)
        _mm_setzero_si128 (),
        (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_andnot_pd (__m256d __W, __mmask8 __U, __m256d __A,
         __m256d __B)
{
  return (__m256d) __builtin_ia32_andnpd256_mask ((__v4df) __A,
        (__v4df) __B,
        (__v4df) __W,
        (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_andnot_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_andnpd256_mask ((__v4df) __A,
        (__v4df) __B,
        (__v4df)
        _mm256_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_andnot_pd (__m128d __W, __mmask8 __U, __m128d __A,
      __m128d __B)
{
  return (__m128d) __builtin_ia32_andnpd128_mask ((__v2df) __A,
        (__v2df) __B,
        (__v2df) __W,
        (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_andnot_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_andnpd128_mask ((__v2df) __A,
        (__v2df) __B,
        (__v2df)
        _mm_setzero_pd (),
        (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_andnot_ps (__m256 __W, __mmask8 __U, __m256 __A,
         __m256 __B)
{
  return (__m256) __builtin_ia32_andnps256_mask ((__v8sf) __A,
       (__v8sf) __B,
       (__v8sf) __W,
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_andnot_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_andnps256_mask ((__v8sf) __A,
       (__v8sf) __B,
       (__v8sf)
       _mm256_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_andnot_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_andnps128_mask ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf) __W,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_andnot_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_andnps128_mask ((__v4sf) __A,
       (__v4sf) __B,
       (__v4sf)
       _mm_setzero_ps (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtps_epi64 (__m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2qq256_mask ((__v4sf) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtps_epi64 (__m256i __W, __mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2qq256_mask ((__v4sf) __A,
          (__v4di) __W,
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtps_epi64 (__mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2qq256_mask ((__v4sf) __A,
          (__v4di)
          _mm256_setzero_si256 (),
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_epi64 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2qq128_mask ((__v4sf) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtps_epi64 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2qq128_mask ((__v4sf) __A,
          (__v2di) __W,
          (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtps_epi64 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2qq128_mask ((__v4sf) __A,
          (__v2di)
          _mm_setzero_si128 (),
          (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtps_epu64 (__m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2uqq256_mask ((__v4sf) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtps_epu64 (__m256i __W, __mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2uqq256_mask ((__v4sf) __A,
           (__v4di) __W,
           (__mmask8) __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtps_epu64 (__mmask8 __U, __m128 __A)
{
  return (__m256i) __builtin_ia32_cvtps2uqq256_mask ((__v4sf) __A,
           (__v4di)
           _mm256_setzero_si256 (),
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtps_epu64 (__m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2uqq128_mask ((__v4sf) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtps_epu64 (__m128i __W, __mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2uqq128_mask ((__v4sf) __A,
           (__v2di) __W,
           (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtps_epu64 (__mmask8 __U, __m128 __A)
{
  return (__m128i) __builtin_ia32_cvtps2uqq128_mask ((__v4sf) __A,
           (__v2di)
           _mm_setzero_si128 (),
           (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi64_ps (__m256i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps256_mask ((__v4di) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_ps (__m128 __W, __mmask8 __U, __m256i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps256_mask ((__v4di) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi64_ps (__mmask8 __U, __m256i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps256_mask ((__v4di) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi64_ps (__m128i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps128_mask ((__v2di) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_ps (__m128 __W, __mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps128_mask ((__v2di) __A,
         (__v4sf) __W,
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi64_ps (__mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtqq2ps128_mask ((__v2di) __A,
         (__v4sf)
         _mm_setzero_ps (),
         (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu64_ps (__m256i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps256_mask ((__v4di) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu64_ps (__m128 __W, __mmask8 __U, __m256i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps256_mask ((__v4di) __A,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu64_ps (__mmask8 __U, __m256i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps256_mask ((__v4di) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu64_ps (__m128i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps128_mask ((__v2di) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) -1);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu64_ps (__m128 __W, __mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps128_mask ((__v2di) __A,
          (__v4sf) __W,
          (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu64_ps (__mmask8 __U, __m128i __A)
{
  return (__m128) __builtin_ia32_cvtuqq2ps128_mask ((__v2di) __A,
          (__v4sf)
          _mm_setzero_ps (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepi64_pd (__m256i __A)
{
  return (__m256d) __builtin_ia32_cvtqq2pd256_mask ((__v4di) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepi64_pd (__m256d __W, __mmask8 __U, __m256i __A)
{
  return (__m256d) __builtin_ia32_cvtqq2pd256_mask ((__v4di) __A,
          (__v4df) __W,
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepi64_pd (__mmask8 __U, __m256i __A)
{
  return (__m256d) __builtin_ia32_cvtqq2pd256_mask ((__v4di) __A,
          (__v4df)
          _mm256_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepi64_pd (__m128i __A)
{
  return (__m128d) __builtin_ia32_cvtqq2pd128_mask ((__v2di) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepi64_pd (__m128d __W, __mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtqq2pd128_mask ((__v2di) __A,
          (__v2df) __W,
          (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepi64_pd (__mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtqq2pd128_mask ((__v2di) __A,
          (__v2df)
          _mm_setzero_pd (),
          (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtepu64_pd (__m256i __A)
{
  return (__m256d) __builtin_ia32_cvtuqq2pd256_mask ((__v4di) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_cvtepu64_pd (__m256d __W, __mmask8 __U, __m256i __A)
{
  return (__m256d) __builtin_ia32_cvtuqq2pd256_mask ((__v4di) __A,
           (__v4df) __W,
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_cvtepu64_pd (__mmask8 __U, __m256i __A)
{
  return (__m256d) __builtin_ia32_cvtuqq2pd256_mask ((__v4di) __A,
           (__v4df)
           _mm256_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_and_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_andpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_and_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_andpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_and_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_andpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_and_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_andpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_and_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_andps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_and_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_andps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_and_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_andps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_and_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_andps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtepu64_pd (__m128i __A)
{
  return (__m128d) __builtin_ia32_cvtuqq2pd128_mask ((__v2di) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) -1);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_cvtepu64_pd (__m128d __W, __mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtuqq2pd128_mask ((__v2di) __A,
           (__v2df) __W,
           (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_cvtepu64_pd (__mmask8 __U, __m128i __A)
{
  return (__m128d) __builtin_ia32_cvtuqq2pd128_mask ((__v2di) __A,
           (__v2df)
           _mm_setzero_pd (),
           (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_xor_pd (__m256d __W, __mmask8 __U, __m256d __A,
      __m256d __B)
{
  return (__m256d) __builtin_ia32_xorpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df) __W,
       (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_xor_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_xorpd256_mask ((__v4df) __A,
       (__v4df) __B,
       (__v4df)
       _mm256_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_xor_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_xorpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df) __W,
       (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_xor_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_xorpd128_mask ((__v2df) __A,
       (__v2df) __B,
       (__v2df)
       _mm_setzero_pd (),
       (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_xor_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_xorps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf) __W,
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_xor_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_xorps256_mask ((__v8sf) __A,
      (__v8sf) __B,
      (__v8sf)
      _mm256_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_xor_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_xorps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf) __W,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_xor_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_xorps128_mask ((__v4sf) __A,
      (__v4sf) __B,
      (__v4sf)
      _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_or_pd (__m256d __W, __mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_orpd256_mask ((__v4df) __A,
      (__v4df) __B,
      (__v4df) __W,
      (__mmask8) __U);
}
extern __inline __m256d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_or_pd (__mmask8 __U, __m256d __A, __m256d __B)
{
  return (__m256d) __builtin_ia32_orpd256_mask ((__v4df) __A,
      (__v4df) __B,
      (__v4df)
      _mm256_setzero_pd (),
      (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_or_pd (__m128d __W, __mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_orpd128_mask ((__v2df) __A,
      (__v2df) __B,
      (__v2df) __W,
      (__mmask8) __U);
}
extern __inline __m128d
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_or_pd (__mmask8 __U, __m128d __A, __m128d __B)
{
  return (__m128d) __builtin_ia32_orpd128_mask ((__v2df) __A,
      (__v2df) __B,
      (__v2df)
      _mm_setzero_pd (),
      (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_or_ps (__m256 __W, __mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_orps256_mask ((__v8sf) __A,
            (__v8sf) __B,
            (__v8sf) __W,
            (__mmask8) __U);
}
extern __inline __m256
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_or_ps (__mmask8 __U, __m256 __A, __m256 __B)
{
  return (__m256) __builtin_ia32_orps256_mask ((__v8sf) __A,
            (__v8sf) __B,
            (__v8sf)
            _mm256_setzero_ps (),
            (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_or_ps (__m128 __W, __mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_orps128_mask ((__v4sf) __A,
            (__v4sf) __B,
            (__v4sf) __W,
            (__mmask8) __U);
}
extern __inline __m128
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_or_ps (__mmask8 __U, __m128 __A, __m128 __B)
{
  return (__m128) __builtin_ia32_orps128_mask ((__v4sf) __A,
            (__v4sf) __B,
            (__v4sf)
            _mm_setzero_ps (),
            (__mmask8) __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movm_epi32 (__mmask8 __A)
{
  return (__m128i) __builtin_ia32_cvtmask2d128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movm_epi32 (__mmask8 __A)
{
  return (__m256i) __builtin_ia32_cvtmask2d256 (__A);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movm_epi64 (__mmask8 __A)
{
  return (__m128i) __builtin_ia32_cvtmask2q128 (__A);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movm_epi64 (__mmask8 __A)
{
  return (__m256i) __builtin_ia32_cvtmask2q256 (__A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movepi32_mask (__m128i __A)
{
  return (__mmask8) __builtin_ia32_cvtd2mask128 ((__v4si) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movepi32_mask (__m256i __A)
{
  return (__mmask8) __builtin_ia32_cvtd2mask256 ((__v8si) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_movepi64_mask (__m128i __A)
{
  return (__mmask8) __builtin_ia32_cvtq2mask128 ((__v2di) __A);
}
extern __inline __mmask8
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_movepi64_mask (__m256i __A)
{
  return (__mmask8) __builtin_ia32_cvtq2mask256 ((__v4di) __A);
}
# 2013 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vldqintrin.h" 3 4
#pragma GCC pop_options
# 62 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512ifmaintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512ifmaintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512ifma")
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_madd52lo_epu64 (__m512i __X, __m512i __Y, __m512i __Z)
{
  return (__m512i) __builtin_ia32_vpmadd52luq512_mask ((__v8di) __X,
             (__v8di) __Y,
             (__v8di) __Z,
             (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_madd52hi_epu64 (__m512i __X, __m512i __Y, __m512i __Z)
{
  return (__m512i) __builtin_ia32_vpmadd52huq512_mask ((__v8di) __X,
             (__v8di) __Y,
             (__v8di) __Z,
             (__mmask8) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_madd52lo_epu64 (__m512i __W, __mmask8 __M, __m512i __X,
       __m512i __Y)
{
  return (__m512i) __builtin_ia32_vpmadd52luq512_mask ((__v8di) __W,
             (__v8di) __X,
             (__v8di) __Y,
             (__mmask8) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_madd52hi_epu64 (__m512i __W, __mmask8 __M, __m512i __X,
       __m512i __Y)
{
  return (__m512i) __builtin_ia32_vpmadd52huq512_mask ((__v8di) __W,
             (__v8di) __X,
             (__v8di) __Y,
             (__mmask8) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_madd52lo_epu64 (__mmask8 __M, __m512i __X, __m512i __Y, __m512i __Z)
{
  return (__m512i) __builtin_ia32_vpmadd52luq512_maskz ((__v8di) __X,
       (__v8di) __Y,
       (__v8di) __Z,
       (__mmask8) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_madd52hi_epu64 (__mmask8 __M, __m512i __X, __m512i __Y, __m512i __Z)
{
  return (__m512i) __builtin_ia32_vpmadd52huq512_maskz ((__v8di) __X,
       (__v8di) __Y,
       (__v8di) __Z,
       (__mmask8) __M);
}
#pragma GCC pop_options
# 64 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512ifmavlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512ifmavlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512ifma,avx512vl")
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_madd52lo_epu64 (__m128i __X, __m128i __Y, __m128i __Z)
{
  return (__m128i) __builtin_ia32_vpmadd52luq128_mask ((__v2di) __X,
             (__v2di) __Y,
             (__v2di) __Z,
             (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_madd52hi_epu64 (__m128i __X, __m128i __Y, __m128i __Z)
{
  return (__m128i) __builtin_ia32_vpmadd52huq128_mask ((__v2di) __X,
             (__v2di) __Y,
             (__v2di) __Z,
             (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_madd52lo_epu64 (__m256i __X, __m256i __Y, __m256i __Z)
{
  return (__m256i) __builtin_ia32_vpmadd52luq256_mask ((__v4di) __X,
             (__v4di) __Y,
             (__v4di) __Z,
             (__mmask8) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_madd52hi_epu64 (__m256i __X, __m256i __Y, __m256i __Z)
{
  return (__m256i) __builtin_ia32_vpmadd52huq256_mask ((__v4di) __X,
             (__v4di) __Y,
             (__v4di) __Z,
             (__mmask8) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_madd52lo_epu64 (__m128i __W, __mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_vpmadd52luq128_mask ((__v2di) __W,
             (__v2di) __X,
             (__v2di) __Y,
             (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_madd52hi_epu64 (__m128i __W, __mmask8 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_vpmadd52huq128_mask ((__v2di) __W,
             (__v2di) __X,
             (__v2di) __Y,
             (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_madd52lo_epu64 (__m256i __W, __mmask8 __M, __m256i __X,
       __m256i __Y)
{
  return (__m256i) __builtin_ia32_vpmadd52luq256_mask ((__v4di) __W,
             (__v4di) __X,
             (__v4di) __Y,
             (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_madd52hi_epu64 (__m256i __W, __mmask8 __M, __m256i __X,
       __m256i __Y)
{
  return (__m256i) __builtin_ia32_vpmadd52huq256_mask ((__v4di) __W,
             (__v4di) __X,
             (__v4di) __Y,
             (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_madd52lo_epu64 (__mmask8 __M, __m128i __X, __m128i __Y, __m128i __Z)
{
  return (__m128i) __builtin_ia32_vpmadd52luq128_maskz ((__v2di) __X,
       (__v2di) __Y,
       (__v2di) __Z,
       (__mmask8) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_madd52hi_epu64 (__mmask8 __M, __m128i __X, __m128i __Y, __m128i __Z)
{
  return (__m128i) __builtin_ia32_vpmadd52huq128_maskz ((__v2di) __X,
       (__v2di) __Y,
       (__v2di) __Z,
       (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_madd52lo_epu64 (__mmask8 __M, __m256i __X, __m256i __Y, __m256i __Z)
{
  return (__m256i) __builtin_ia32_vpmadd52luq256_maskz ((__v4di) __X,
       (__v4di) __Y,
       (__v4di) __Z,
       (__mmask8) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_madd52hi_epu64 (__mmask8 __M, __m256i __X, __m256i __Y, __m256i __Z)
{
  return (__m256i) __builtin_ia32_vpmadd52huq256_maskz ((__v4di) __X,
       (__v4di) __Y,
       (__v4di) __Z,
       (__mmask8) __M);
}
#pragma GCC pop_options
# 66 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmiintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmiintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vbmi")
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_multishift_epi64_epi8 (__m512i __W, __mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_vpmultishiftqb512_mask ((__v64qi) __X,
         (__v64qi) __Y,
         (__v64qi) __W,
         (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_multishift_epi64_epi8 (__mmask64 __M, __m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_vpmultishiftqb512_mask ((__v64qi) __X,
         (__v64qi) __Y,
         (__v64qi)
         _mm512_setzero_si512 (),
         (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_multishift_epi64_epi8 (__m512i __X, __m512i __Y)
{
  return (__m512i) __builtin_ia32_vpmultishiftqb512_mask ((__v64qi) __X,
         (__v64qi) __Y,
         (__v64qi)
         _mm512_undefined_epi32 (),
         (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutexvar_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarqi512_mask ((__v64qi) __B,
           (__v64qi) __A,
           (__v64qi)
           _mm512_undefined_epi32 (),
           (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutexvar_epi8 (__mmask64 __M, __m512i __A,
    __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarqi512_mask ((__v64qi) __B,
           (__v64qi) __A,
           (__v64qi)
           _mm512_setzero_si512(),
           (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutexvar_epi8 (__m512i __W, __mmask64 __M, __m512i __A,
          __m512i __B)
{
  return (__m512i) __builtin_ia32_permvarqi512_mask ((__v64qi) __B,
           (__v64qi) __A,
           (__v64qi) __W,
           (__mmask64) __M);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_permutex2var_epi8 (__m512i __A, __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varqi512_mask ((__v64qi) __I
                 ,
       (__v64qi) __A,
       (__v64qi) __B,
       (__mmask64) -1);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_permutex2var_epi8 (__m512i __A, __mmask64 __U,
    __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varqi512_mask ((__v64qi) __I
                 ,
       (__v64qi) __A,
       (__v64qi) __B,
       (__mmask64)
       __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask2_permutex2var_epi8 (__m512i __A, __m512i __I,
     __mmask64 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermi2varqi512_mask ((__v64qi) __A,
       (__v64qi) __I
                 ,
       (__v64qi) __B,
       (__mmask64)
       __U);
}
extern __inline __m512i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_permutex2var_epi8 (__mmask64 __U, __m512i __A,
     __m512i __I, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpermt2varqi512_maskz ((__v64qi) __I
                  ,
        (__v64qi) __A,
        (__v64qi) __B,
        (__mmask64)
        __U);
}
#pragma GCC pop_options
# 68 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmivlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmivlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vbmi,avx512vl")
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_multishift_epi64_epi8 (__m256i __W, __mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_vpmultishiftqb256_mask ((__v32qi) __X,
         (__v32qi) __Y,
         (__v32qi) __W,
         (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_multishift_epi64_epi8 (__mmask32 __M, __m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_vpmultishiftqb256_mask ((__v32qi) __X,
         (__v32qi) __Y,
         (__v32qi)
         _mm256_setzero_si256 (),
         (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_multishift_epi64_epi8 (__m256i __X, __m256i __Y)
{
  return (__m256i) __builtin_ia32_vpmultishiftqb256_mask ((__v32qi) __X,
         (__v32qi) __Y,
         (__v32qi)
         _mm256_undefined_si256 (),
         (__mmask32) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_multishift_epi64_epi8 (__m128i __W, __mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_vpmultishiftqb128_mask ((__v16qi) __X,
         (__v16qi) __Y,
         (__v16qi) __W,
         (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_multishift_epi64_epi8 (__mmask16 __M, __m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_vpmultishiftqb128_mask ((__v16qi) __X,
         (__v16qi) __Y,
         (__v16qi)
         _mm_setzero_si128 (),
         (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_multishift_epi64_epi8 (__m128i __X, __m128i __Y)
{
  return (__m128i) __builtin_ia32_vpmultishiftqb128_mask ((__v16qi) __X,
         (__v16qi) __Y,
         (__v16qi)
         _mm_undefined_si128 (),
         (__mmask16) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutexvar_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarqi256_mask ((__v32qi) __B,
           (__v32qi) __A,
           (__v32qi)
           _mm256_undefined_si256 (),
           (__mmask32) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutexvar_epi8 (__mmask32 __M, __m256i __A,
    __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarqi256_mask ((__v32qi) __B,
           (__v32qi) __A,
           (__v32qi)
           _mm256_setzero_si256 (),
           (__mmask32) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutexvar_epi8 (__m256i __W, __mmask32 __M, __m256i __A,
          __m256i __B)
{
  return (__m256i) __builtin_ia32_permvarqi256_mask ((__v32qi) __B,
           (__v32qi) __A,
           (__v32qi) __W,
           (__mmask32) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutexvar_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarqi128_mask ((__v16qi) __B,
           (__v16qi) __A,
           (__v16qi)
           _mm_undefined_si128 (),
           (__mmask16) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutexvar_epi8 (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarqi128_mask ((__v16qi) __B,
           (__v16qi) __A,
           (__v16qi)
           _mm_setzero_si128 (),
           (__mmask16) __M);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutexvar_epi8 (__m128i __W, __mmask16 __M, __m128i __A,
       __m128i __B)
{
  return (__m128i) __builtin_ia32_permvarqi128_mask ((__v16qi) __B,
           (__v16qi) __A,
           (__v16qi) __W,
           (__mmask16) __M);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_permutex2var_epi8 (__m256i __A, __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varqi256_mask ((__v32qi) __I
                 ,
       (__v32qi) __A,
       (__v32qi) __B,
       (__mmask32) -1);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_permutex2var_epi8 (__m256i __A, __mmask32 __U,
    __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varqi256_mask ((__v32qi) __I
                 ,
       (__v32qi) __A,
       (__v32qi) __B,
       (__mmask32)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask2_permutex2var_epi8 (__m256i __A, __m256i __I,
     __mmask32 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermi2varqi256_mask ((__v32qi) __A,
       (__v32qi) __I
                 ,
       (__v32qi) __B,
       (__mmask32)
       __U);
}
extern __inline __m256i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_permutex2var_epi8 (__mmask32 __U, __m256i __A,
     __m256i __I, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpermt2varqi256_maskz ((__v32qi) __I
                  ,
        (__v32qi) __A,
        (__v32qi) __B,
        (__mmask32)
        __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_permutex2var_epi8 (__m128i __A, __m128i __I, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varqi128_mask ((__v16qi) __I
                 ,
       (__v16qi) __A,
       (__v16qi) __B,
       (__mmask16) -1);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_permutex2var_epi8 (__m128i __A, __mmask16 __U, __m128i __I,
        __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varqi128_mask ((__v16qi) __I
                 ,
       (__v16qi) __A,
       (__v16qi) __B,
       (__mmask16)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask2_permutex2var_epi8 (__m128i __A, __m128i __I, __mmask16 __U,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermi2varqi128_mask ((__v16qi) __A,
       (__v16qi) __I
                 ,
       (__v16qi) __B,
       (__mmask16)
       __U);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_permutex2var_epi8 (__mmask16 __U, __m128i __A, __m128i __I,
         __m128i __B)
{
  return (__m128i) __builtin_ia32_vpermt2varqi128_maskz ((__v16qi) __I
                  ,
        (__v16qi) __A,
        (__v16qi) __B,
        (__mmask16)
        __U);
}
#pragma GCC pop_options
# 70 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx5124fmapsintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx5124fmapsintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx5124fmaps")
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_4fmadd_ps (__m512 __A, __m512 __B, __m512 __C,
    __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fmaddps ((__v16sf) __B,
        (__v16sf) __C,
        (__v16sf) __D,
        (__v16sf) __E,
        (__v16sf) __A,
        (const __v4sf *) __F);
}
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_4fmadd_ps (__m512 __A, __mmask16 __U, __m512 __B,
         __m512 __C, __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fmaddps_mask ((__v16sf) __B,
      (__v16sf) __C,
      (__v16sf) __D,
      (__v16sf) __E,
      (__v16sf) __A,
      (const __v4sf *) __F,
      (__v16sf) __A,
      (__mmask16) __U);
}
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_4fmadd_ps (__mmask16 __U,
   __m512 __A, __m512 __B, __m512 __C,
   __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fmaddps_mask ((__v16sf) __B,
      (__v16sf) __C,
      (__v16sf) __D,
      (__v16sf) __E,
      (__v16sf) __A,
      (const __v4sf *) __F,
      (__v16sf) _mm512_setzero_ps (),
      (__mmask16) __U);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_4fmadd_ss (__m128 __A, __m128 __B, __m128 __C,
        __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fmaddss ((__v4sf) __B,
        (__v4sf) __C,
        (__v4sf) __D,
        (__v4sf) __E,
        (__v4sf) __A,
        (const __v4sf *) __F);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_4fmadd_ss (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C,
      __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fmaddss_mask ((__v4sf) __B,
      (__v4sf) __C,
      (__v4sf) __D,
      (__v4sf) __E,
      (__v4sf) __A,
      (const __v4sf *) __F,
      (__v4sf) __A,
      (__mmask8) __U);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_4fmadd_ss (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C,
       __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fmaddss_mask ((__v4sf) __B,
      (__v4sf) __C,
      (__v4sf) __D,
      (__v4sf) __E,
      (__v4sf) __A,
      (const __v4sf *) __F,
      (__v4sf) _mm_setzero_ps (),
      (__mmask8) __U);
}
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_4fnmadd_ps (__m512 __A, __m512 __B, __m512 __C,
     __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fnmaddps ((__v16sf) __B,
         (__v16sf) __C,
         (__v16sf) __D,
         (__v16sf) __E,
         (__v16sf) __A,
         (const __v4sf *) __F);
}
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_4fnmadd_ps (__m512 __A, __mmask16 __U, __m512 __B,
   __m512 __C, __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fnmaddps_mask ((__v16sf) __B,
       (__v16sf) __C,
       (__v16sf) __D,
       (__v16sf) __E,
       (__v16sf) __A,
       (const __v4sf *) __F,
       (__v16sf) __A,
       (__mmask16) __U);
}
extern __inline __m512
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_4fnmadd_ps (__mmask16 __U,
    __m512 __A, __m512 __B, __m512 __C,
    __m512 __D, __m512 __E, __m128 *__F)
{
  return (__m512) __builtin_ia32_4fnmaddps_mask ((__v16sf) __B,
       (__v16sf) __C,
       (__v16sf) __D,
       (__v16sf) __E,
       (__v16sf) __A,
       (const __v4sf *) __F,
       (__v16sf) _mm512_setzero_ps (),
       (__mmask16) __U);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_4fnmadd_ss (__m128 __A, __m128 __B, __m128 __C,
  __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fnmaddss ((__v4sf) __B,
         (__v4sf) __C,
         (__v4sf) __D,
         (__v4sf) __E,
         (__v4sf) __A,
         (const __v4sf *) __F);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_4fnmadd_ss (__m128 __A, __mmask8 __U, __m128 __B, __m128 __C,
       __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fnmaddss_mask ((__v4sf) __B,
       (__v4sf) __C,
       (__v4sf) __D,
       (__v4sf) __E,
       (__v4sf) __A,
       (const __v4sf *) __F,
       (__v4sf) __A,
       (__mmask8) __U);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_4fnmadd_ss (__mmask8 __U, __m128 __A, __m128 __B, __m128 __C,
        __m128 __D, __m128 __E, __m128 *__F)
{
  return (__m128) __builtin_ia32_4fnmaddss_mask ((__v4sf) __B,
       (__v4sf) __C,
       (__v4sf) __D,
       (__v4sf) __E,
       (__v4sf) __A,
       (const __v4sf *) __F,
       (__v4sf) _mm_setzero_ps (),
       (__mmask8) __U);
}
#pragma GCC pop_options
# 72 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx5124vnniwintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx5124vnniwintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx5124vnniw")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_4dpwssd_epi32 (__m512i __A, __m512i __B, __m512i __C,
        __m512i __D, __m512i __E, __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssd ((__v16si) __B,
          (__v16si) __C,
          (__v16si) __D,
          (__v16si) __E,
          (__v16si) __A,
          (const __v4si *) __F);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_4dpwssd_epi32 (__m512i __A, __mmask16 __U, __m512i __B,
      __m512i __C, __m512i __D, __m512i __E,
      __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssd_mask ((__v16si) __B,
        (__v16si) __C,
        (__v16si) __D,
        (__v16si) __E,
        (__v16si) __A,
        (const __v4si *) __F,
        (__v16si) __A,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_4dpwssd_epi32 (__mmask16 __U, __m512i __A, __m512i __B,
       __m512i __C, __m512i __D, __m512i __E,
       __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssd_mask ((__v16si) __B,
        (__v16si) __C,
        (__v16si) __D,
        (__v16si) __E,
        (__v16si) __A,
        (const __v4si *) __F,
        (__v16si) _mm512_setzero_ps (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_4dpwssds_epi32 (__m512i __A, __m512i __B, __m512i __C,
         __m512i __D, __m512i __E, __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssds ((__v16si) __B,
           (__v16si) __C,
           (__v16si) __D,
           (__v16si) __E,
           (__v16si) __A,
           (const __v4si *) __F);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_4dpwssds_epi32 (__m512i __A, __mmask16 __U, __m512i __B,
       __m512i __C, __m512i __D, __m512i __E,
       __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssds_mask ((__v16si) __B,
         (__v16si) __C,
         (__v16si) __D,
         (__v16si) __E,
         (__v16si) __A,
         (const __v4si *) __F,
         (__v16si) __A,
         (__mmask16) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_4dpwssds_epi32 (__mmask16 __U, __m512i __A, __m512i __B,
        __m512i __C, __m512i __D, __m512i __E,
        __m128i *__F)
{
  return (__m512i) __builtin_ia32_vp4dpwssds_mask ((__v16si) __B,
         (__v16si) __C,
         (__v16si) __D,
         (__v16si) __E,
         (__v16si) __A,
         (const __v4si *) __F,
         (__v16si) _mm512_setzero_ps (),
         (__mmask16) __U);
}
#pragma GCC pop_options
# 74 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vpopcntdqintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vpopcntdqintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vpopcntdq")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_popcnt_epi32 (__m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountd_v16si ((__v16si) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_popcnt_epi32 (__m512i __A, __mmask16 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpopcountd_v16si_mask ((__v16si) __A,
        (__v16si) __B,
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_popcnt_epi32 (__mmask16 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountd_v16si_mask ((__v16si) __A,
        (__v16si)
        _mm512_setzero_si512 (),
        (__mmask16) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_popcnt_epi64 (__m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountq_v8di ((__v8di) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_popcnt_epi64 (__m512i __A, __mmask8 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpopcountq_v8di_mask ((__v8di) __A,
       (__v8di) __B,
       (__mmask8) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_popcnt_epi64 (__mmask8 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountq_v8di_mask ((__v8di) __A,
       (__v8di)
       _mm512_setzero_si512 (),
       (__mmask8) __U);
}
#pragma GCC pop_options
# 76 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2intrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2intrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vbmi2")
# 200 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2intrin.h" 3 4
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shrdv_epi16 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshrdv_v32hi ((__v32hi)__A, (__v32hi) __B,
        (__v32hi) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shrdv_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshrdv_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shrdv_epi32 (__m512i __A, __mmask16 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shrdv_epi32 (__mmask16 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shrdv_epi64 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshrdv_v8di ((__v8di)__A, (__v8di) __B,
        (__v8di) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shrdv_epi64 (__m512i __A, __mmask8 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v8di_mask ((__v8di)__A, (__v8di) __C,
      (__v8di) __D, (__mmask8)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shrdv_epi64 (__mmask8 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v8di_maskz ((__v8di)__B, (__v8di) __C,
       (__v8di) __D, (__mmask8)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shldv_epi16 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshldv_v32hi ((__v32hi)__A, (__v32hi) __B,
        (__v32hi) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shldv_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshldv_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shldv_epi32 (__m512i __A, __mmask16 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shldv_epi32 (__mmask16 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_shldv_epi64 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpshldv_v8di ((__v8di)__A, (__v8di) __B,
        (__v8di) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shldv_epi64 (__m512i __A, __mmask8 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v8di_mask ((__v8di)__A, (__v8di) __C,
      (__v8di) __D, (__mmask8)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shldv_epi64 (__mmask8 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v8di_maskz ((__v8di)__B, (__v8di) __C,
      (__v8di) __D, (__mmask8)__A);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("avx512vbmi2,avx512bw")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_epi8 (__m512i __A, __mmask64 __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_compressqi512_mask ((__v64qi)__C,
      (__v64qi)__A, (__mmask64)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_epi8 (__mmask64 __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_compressqi512_mask ((__v64qi)__B,
   (__v64qi)_mm512_setzero_si512 (), (__mmask64)__A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_epi8 (void * __A, __mmask64 __B, __m512i __C)
{
  __builtin_ia32_compressstoreuqi512_mask ((__v64qi *) __A, (__v64qi) __C,
       (__mmask64) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compress_epi16 (__m512i __A, __mmask32 __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_compresshi512_mask ((__v32hi)__C,
      (__v32hi)__A, (__mmask32)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_compress_epi16 (__mmask32 __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_compresshi512_mask ((__v32hi)__B,
   (__v32hi)_mm512_setzero_si512 (), (__mmask32)__A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_compressstoreu_epi16 (void * __A, __mmask32 __B, __m512i __C)
{
  __builtin_ia32_compressstoreuhi512_mask ((__v32hi *) __A, (__v32hi) __C,
       (__mmask32) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_epi8 (__m512i __A, __mmask64 __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_expandqi512_mask ((__v64qi) __C,
          (__v64qi) __A,
          (__mmask64) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_epi8 (__mmask64 __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_expandqi512_maskz ((__v64qi) __B,
   (__v64qi) _mm512_setzero_si512 (), (__mmask64) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_epi8 (__m512i __A, __mmask64 __B, const void * __C)
{
  return (__m512i) __builtin_ia32_expandloadqi512_mask ((const __v64qi *) __C,
     (__v64qi) __A, (__mmask64) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_epi8 (__mmask64 __A, const void * __B)
{
  return (__m512i) __builtin_ia32_expandloadqi512_maskz ((const __v64qi *) __B,
   (__v64qi) _mm512_setzero_si512 (), (__mmask64) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expand_epi16 (__m512i __A, __mmask32 __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_expandhi512_mask ((__v32hi) __C,
          (__v32hi) __A,
          (__mmask32) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expand_epi16 (__mmask32 __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_expandhi512_maskz ((__v32hi) __B,
   (__v32hi) _mm512_setzero_si512 (), (__mmask32) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_expandloadu_epi16 (__m512i __A, __mmask32 __B, const void * __C)
{
  return (__m512i) __builtin_ia32_expandloadhi512_mask ((const __v32hi *) __C,
     (__v32hi) __A, (__mmask32) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_expandloadu_epi16 (__mmask32 __A, const void * __B)
{
  return (__m512i) __builtin_ia32_expandloadhi512_maskz ((const __v32hi *) __B,
   (__v32hi) _mm512_setzero_si512 (), (__mmask32) __A);
}
# 491 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2intrin.h" 3 4
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shrdv_epi16 (__m512i __A, __mmask32 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v32hi_mask ((__v32hi)__A,
    (__v32hi) __C, (__v32hi) __D, (__mmask32)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shrdv_epi16 (__mmask32 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshrdv_v32hi_maskz ((__v32hi)__B,
    (__v32hi) __C, (__v32hi) __D, (__mmask32)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_shldv_epi16 (__m512i __A, __mmask32 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v32hi_mask ((__v32hi)__A,
    (__v32hi) __C, (__v32hi) __D, (__mmask32)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_shldv_epi16 (__mmask32 __A, __m512i __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpshldv_v32hi_maskz ((__v32hi)__B,
    (__v32hi) __C, (__v32hi) __D, (__mmask32)__A);
}
#pragma GCC pop_options
# 78 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2vlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2vlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vbmi2,avx512vl")
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_epi8 (__m128i __A, __mmask16 __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_compressqi128_mask ((__v16qi)__C,
      (__v16qi)__A, (__mmask16)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_epi8 (__mmask16 __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_compressqi128_mask ((__v16qi) __B,
   (__v16qi) _mm_setzero_si128 (), (__mmask16) __A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_epi16 (void * __A, __mmask16 __B, __m256i __C)
{
  __builtin_ia32_compressstoreuhi256_mask ((__v16hi *) __A, (__v16hi) __C,
       (__mmask16) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compress_epi16 (__m128i __A, __mmask8 __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_compresshi128_mask ((__v8hi)__C, (__v8hi)__A,
        (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_compress_epi16 (__mmask8 __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_compresshi128_mask ((__v8hi) __B,
    (__v8hi) _mm_setzero_si128 (), (__mmask8) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_epi16 (__m256i __A, __mmask16 __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_compresshi256_mask ((__v16hi)__C,
      (__v16hi)__A, (__mmask16)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_epi16 (__mmask16 __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_compresshi256_mask ((__v16hi) __B,
   (__v16hi) _mm256_setzero_si256 (), (__mmask16) __A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_epi8 (void * __A, __mmask16 __B, __m128i __C)
{
  __builtin_ia32_compressstoreuqi128_mask ((__v16qi *) __A, (__v16qi) __C,
       (__mmask16) __B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_compressstoreu_epi16 (void * __A, __mmask8 __B, __m128i __C)
{
  __builtin_ia32_compressstoreuhi128_mask ((__v8hi *) __A, (__v8hi) __C,
       (__mmask8) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_epi8 (__m128i __A, __mmask16 __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_expandqi128_mask ((__v16qi) __C,
          (__v16qi) __A,
          (__mmask16) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_epi8 (__mmask16 __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_expandqi128_maskz ((__v16qi) __B,
   (__v16qi) _mm_setzero_si128 (), (__mmask16) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_epi8 (__m128i __A, __mmask16 __B, const void * __C)
{
  return (__m128i) __builtin_ia32_expandloadqi128_mask ((const __v16qi *) __C,
     (__v16qi) __A, (__mmask16) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_epi8 (__mmask16 __A, const void * __B)
{
  return (__m128i) __builtin_ia32_expandloadqi128_maskz ((const __v16qi *) __B,
   (__v16qi) _mm_setzero_si128 (), (__mmask16) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expand_epi16 (__m128i __A, __mmask8 __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_expandhi128_mask ((__v8hi) __C,
          (__v8hi) __A,
          (__mmask8) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expand_epi16 (__mmask8 __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_expandhi128_maskz ((__v8hi) __B,
    (__v8hi) _mm_setzero_si128 (), (__mmask8) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_expandloadu_epi16 (__m128i __A, __mmask8 __B, const void * __C)
{
  return (__m128i) __builtin_ia32_expandloadhi128_mask ((const __v8hi *) __C,
      (__v8hi) __A, (__mmask8) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_expandloadu_epi16 (__mmask8 __A, const void * __B)
{
  return (__m128i) __builtin_ia32_expandloadhi128_maskz ((const __v8hi *) __B,
    (__v8hi) _mm_setzero_si128 (), (__mmask8) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_epi16 (__m256i __A, __mmask16 __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_expandhi256_mask ((__v16hi) __C,
          (__v16hi) __A,
          (__mmask16) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_epi16 (__mmask16 __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_expandhi256_maskz ((__v16hi) __B,
   (__v16hi) _mm256_setzero_si256 (), (__mmask16) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_epi16 (__m256i __A, __mmask16 __B, const void * __C)
{
  return (__m256i) __builtin_ia32_expandloadhi256_mask ((const __v16hi *) __C,
     (__v16hi) __A, (__mmask16) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_epi16 (__mmask16 __A, const void * __B)
{
  return (__m256i) __builtin_ia32_expandloadhi256_maskz ((const __v16hi *) __B,
   (__v16hi) _mm256_setzero_si256 (), (__mmask16) __A);
}
# 621 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vbmi2vlintrin.h" 3 4
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shrdv_epi16 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshrdv_v16hi ((__v16hi)__A, (__v16hi) __B,
        (__v16hi) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shrdv_epi16 (__m256i __A, __mmask16 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v16hi_mask ((__v16hi)__A,
    (__v16hi) __C, (__v16hi) __D, (__mmask16)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shrdv_epi16 (__mmask16 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v16hi_maskz ((__v16hi)__B,
    (__v16hi) __C, (__v16hi) __D, (__mmask16)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shrdv_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshrdv_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shrdv_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v8si_mask ((__v8si)__A, (__v8si) __C,
      (__v8si) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shrdv_epi32 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v8si_maskz ((__v8si)__B, (__v8si) __C,
       (__v8si) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shrdv_epi64 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshrdv_v4di ((__v4di)__A, (__v4di) __B,
        (__v4di) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shrdv_epi64 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v4di_mask ((__v4di)__A, (__v4di) __C,
      (__v4di) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shrdv_epi64 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshrdv_v4di_maskz ((__v4di)__B, (__v4di) __C,
       (__v4di) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shrdv_epi16 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshrdv_v8hi ((__v8hi)__A, (__v8hi) __B,
        (__v8hi) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shrdv_epi16 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v8hi_mask ((__v8hi)__A, (__v8hi) __C,
      (__v8hi) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shrdv_epi16 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v8hi_maskz ((__v8hi)__B, (__v8hi) __C,
       (__v8hi) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shrdv_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshrdv_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shrdv_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v4si_mask ((__v4si)__A, (__v4si) __C,
      (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shrdv_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v4si_maskz ((__v4si)__B, (__v4si) __C,
       (__v4si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shrdv_epi64 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshrdv_v2di ((__v2di)__A, (__v2di) __B,
        (__v2di) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shrdv_epi64 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v2di_mask ((__v2di)__A, (__v2di) __C,
      (__v2di) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shrdv_epi64 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshrdv_v2di_maskz ((__v2di)__B, (__v2di) __C,
       (__v2di) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shldv_epi16 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshldv_v16hi ((__v16hi)__A, (__v16hi) __B,
        (__v16hi) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shldv_epi16 (__m256i __A, __mmask16 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v16hi_mask ((__v16hi)__A,
    (__v16hi) __C, (__v16hi) __D, (__mmask16)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shldv_epi16 (__mmask16 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v16hi_maskz ((__v16hi)__B,
    (__v16hi) __C, (__v16hi) __D, (__mmask16)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shldv_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshldv_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shldv_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v8si_mask ((__v8si)__A, (__v8si) __C,
      (__v8si) __D, (__mmask8)__B) ;
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shldv_epi32 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v8si_maskz ((__v8si)__B, (__v8si) __C,
      (__v8si) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_shldv_epi64 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpshldv_v4di ((__v4di)__A, (__v4di) __B,
        (__v4di) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_shldv_epi64 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v4di_mask ((__v4di)__A, (__v4di) __C,
      (__v4di) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_shldv_epi64 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpshldv_v4di_maskz ((__v4di)__B, (__v4di) __C,
       (__v4di) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shldv_epi16 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshldv_v8hi ((__v8hi)__A, (__v8hi) __B,
        (__v8hi) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shldv_epi16 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v8hi_mask ((__v8hi)__A, (__v8hi) __C,
      (__v8hi) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shldv_epi16 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v8hi_maskz ((__v8hi)__B, (__v8hi) __C,
       (__v8hi) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shldv_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshldv_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shldv_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v4si_mask ((__v4si)__A, (__v4si) __C,
      (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shldv_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v4si_maskz ((__v4si)__B, (__v4si) __C,
       (__v4si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_shldv_epi64 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpshldv_v2di ((__v2di)__A, (__v2di) __B,
        (__v2di) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_shldv_epi64 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v2di_mask ((__v2di)__A, (__v2di) __C,
      (__v2di) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_shldv_epi64 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpshldv_v2di_maskz ((__v2di)__B, (__v2di) __C,
      (__v2di) __D, (__mmask8)__A);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("avx512vbmi2,avx512vl,avx512bw")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compress_epi8 (__m256i __A, __mmask32 __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_compressqi256_mask ((__v32qi)__C,
      (__v32qi)__A, (__mmask32)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_compress_epi8 (__mmask32 __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_compressqi256_mask ((__v32qi) __B,
   (__v32qi) _mm256_setzero_si256 (), (__mmask32) __A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_compressstoreu_epi8 (void * __A, __mmask32 __B, __m256i __C)
{
  __builtin_ia32_compressstoreuqi256_mask ((__v32qi *) __A, (__v32qi) __C,
       (__mmask32) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expand_epi8 (__m256i __A, __mmask32 __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_expandqi256_mask ((__v32qi) __C,
          (__v32qi) __A,
          (__mmask32) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expand_epi8 (__mmask32 __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_expandqi256_maskz ((__v32qi) __B,
   (__v32qi) _mm256_setzero_si256 (), (__mmask32) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_expandloadu_epi8 (__m256i __A, __mmask32 __B, const void * __C)
{
  return (__m256i) __builtin_ia32_expandloadqi256_mask ((const __v32qi *) __C,
     (__v32qi) __A, (__mmask32) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_expandloadu_epi8 (__mmask32 __A, const void * __B)
{
  return (__m256i) __builtin_ia32_expandloadqi256_maskz ((const __v32qi *) __B,
   (__v32qi) _mm256_setzero_si256 (), (__mmask32) __A);
}
#pragma GCC pop_options
# 80 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vnniintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vnniintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vnni")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_dpbusd_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpdpbusd_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_dpbusd_epi32 (__m512i __A, __mmask16 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpbusd_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_dpbusd_epi32 (__mmask16 __A, __m512i __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpbusd_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_dpbusds_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpdpbusds_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_dpbusds_epi32 (__m512i __A, __mmask16 __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpbusds_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_dpbusds_epi32 (__mmask16 __A, __m512i __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpbusds_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_dpwssd_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpdpwssd_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_dpwssd_epi32 (__m512i __A, __mmask16 __B, __m512i __C, __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpwssd_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_dpwssd_epi32 (__mmask16 __A, __m512i __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpwssd_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_dpwssds_epi32 (__m512i __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vpdpwssds_v16si ((__v16si)__A, (__v16si) __B,
        (__v16si) __C);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_dpwssds_epi32 (__m512i __A, __mmask16 __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpwssds_v16si_mask ((__v16si)__A,
    (__v16si) __C, (__v16si) __D, (__mmask16)__B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_dpwssds_epi32 (__mmask16 __A, __m512i __B, __m512i __C,
       __m512i __D)
{
  return (__m512i)__builtin_ia32_vpdpwssds_v16si_maskz ((__v16si)__B,
    (__v16si) __C, (__v16si) __D, (__mmask16)__A);
}
#pragma GCC pop_options
# 82 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vnnivlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vnnivlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vnni,avx512vl")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_dpbusd_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpdpbusd_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_dpbusd_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpbusd_v8si_mask ((__v8si)__A, (__v8si) __C,
      (__v8si) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_dpbusd_epi32 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpbusd_v8si_maskz ((__v8si)__B,
    (__v8si) __C, (__v8si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_dpbusd_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpdpbusd_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_dpbusd_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpbusd_v4si_mask ((__v4si)__A, (__v4si) __C,
      (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_dpbusd_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpbusd_v4si_maskz ((__v4si)__B,
    (__v4si) __C, (__v4si) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_dpbusds_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpdpbusds_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_dpbusds_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpbusds_v8si_mask ((__v8si)__A,
    (__v8si) __C, (__v8si) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_dpbusds_epi32 (__mmask8 __A, __m256i __B, __m256i __C,
        __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpbusds_v8si_maskz ((__v8si)__B,
    (__v8si) __C, (__v8si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_dpbusds_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpdpbusds_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_dpbusds_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpbusds_v4si_mask ((__v4si)__A,
    (__v4si) __C, (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_dpbusds_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpbusds_v4si_maskz ((__v4si)__B,
    (__v4si) __C, (__v4si) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_dpwssd_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpdpwssd_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_dpwssd_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpwssd_v8si_mask ((__v8si)__A, (__v8si) __C,
      (__v8si) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_dpwssd_epi32 (__mmask8 __A, __m256i __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpwssd_v8si_maskz ((__v8si)__B,
    (__v8si) __C, (__v8si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_dpwssd_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpdpwssd_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_dpwssd_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpwssd_v4si_mask ((__v4si)__A, (__v4si) __C,
      (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_dpwssd_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpwssd_v4si_maskz ((__v4si)__B,
    (__v4si) __C, (__v4si) __D, (__mmask8)__A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_dpwssds_epi32 (__m256i __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vpdpwssds_v8si ((__v8si)__A, (__v8si) __B,
        (__v8si) __C);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_dpwssds_epi32 (__m256i __A, __mmask8 __B, __m256i __C, __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpwssds_v8si_mask ((__v8si)__A,
    (__v8si) __C, (__v8si) __D, (__mmask8)__B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_dpwssds_epi32 (__mmask8 __A, __m256i __B, __m256i __C,
       __m256i __D)
{
  return (__m256i)__builtin_ia32_vpdpwssds_v8si_maskz ((__v8si)__B,
    (__v8si) __C, (__v8si) __D, (__mmask8)__A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_dpwssds_epi32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vpdpwssds_v4si ((__v4si)__A, (__v4si) __B,
        (__v4si) __C);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_dpwssds_epi32 (__m128i __A, __mmask8 __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpwssds_v4si_mask ((__v4si)__A,
    (__v4si) __C, (__v4si) __D, (__mmask8)__B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_dpwssds_epi32 (__mmask8 __A, __m128i __B, __m128i __C, __m128i __D)
{
  return (__m128i)__builtin_ia32_vpdpwssds_v4si_maskz ((__v4si)__B,
    (__v4si) __C, (__v4si) __D, (__mmask8)__A);
}
#pragma GCC pop_options
# 84 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vpopcntdqvlintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512vpopcntdqvlintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512vpopcntdq,avx512vl")
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_epi32 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountd_v4si ((__v4si) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_popcnt_epi32 (__m128i __A, __mmask16 __U, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpopcountd_v4si_mask ((__v4si) __A,
        (__v4si) __B,
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_popcnt_epi32 (__mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountd_v4si_mask ((__v4si) __A,
        (__v4si)
        _mm_setzero_si128 (),
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_popcnt_epi32 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountd_v8si ((__v8si) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_popcnt_epi32 (__m256i __A, __mmask16 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpopcountd_v8si_mask ((__v8si) __A,
        (__v8si) __B,
        (__mmask16) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_popcnt_epi32 (__mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountd_v8si_mask ((__v8si) __A,
      (__v8si)
      _mm256_setzero_si256 (),
      (__mmask16) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_epi64 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountq_v2di ((__v2di) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_popcnt_epi64 (__m128i __A, __mmask8 __U, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpopcountq_v2di_mask ((__v2di) __A,
       (__v2di) __B,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_popcnt_epi64 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountq_v2di_mask ((__v2di) __A,
       (__v2di)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_popcnt_epi64 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountq_v4di ((__v4di) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_popcnt_epi64 (__m256i __A, __mmask8 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpopcountq_v4di_mask ((__v4di) __A,
       (__v4di) __B,
       (__mmask8) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_popcnt_epi64 (__mmask8 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountq_v4di_mask ((__v4di) __A,
      (__v4di)
      _mm256_setzero_si256 (),
      (__mmask8) __U);
}
#pragma GCC pop_options
# 86 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512bitalgintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/avx512bitalgintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("avx512bitalg")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_popcnt_epi8 (__m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountb_v64qi ((__v64qi) __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_popcnt_epi16 (__m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountw_v32hi ((__v32hi) __A);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("avx512bitalg,avx512bw")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_popcnt_epi8 (__m512i __A, __mmask64 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpopcountb_v64qi_mask ((__v64qi) __A,
        (__v64qi) __B,
        (__mmask64) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_popcnt_epi8 (__mmask64 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountb_v64qi_mask ((__v64qi) __A,
      (__v64qi)
      _mm512_setzero_si512 (),
      (__mmask64) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_popcnt_epi16 (__m512i __A, __mmask32 __U, __m512i __B)
{
  return (__m512i) __builtin_ia32_vpopcountw_v32hi_mask ((__v32hi) __A,
       (__v32hi) __B,
       (__mmask32) __U);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_popcnt_epi16 (__mmask32 __U, __m512i __A)
{
  return (__m512i) __builtin_ia32_vpopcountw_v32hi_mask ((__v32hi) __A,
      (__v32hi)
      _mm512_setzero_si512 (),
      (__mmask32) __U);
}
extern __inline __mmask64
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_bitshuffle_epi64_mask (__m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_vpshufbitqmb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__mmask64) -1);
}
extern __inline __mmask64
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_bitshuffle_epi64_mask (__mmask64 __M, __m512i __A, __m512i __B)
{
  return (__mmask64) __builtin_ia32_vpshufbitqmb512_mask ((__v64qi) __A,
       (__v64qi) __B,
       (__mmask64) __M);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("avx512bitalg,avx512vl,avx512bw")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_popcnt_epi8 (__m256i __A, __mmask32 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpopcountb_v32qi_mask ((__v32qi) __A,
        (__v32qi) __B,
        (__mmask32) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_popcnt_epi8 (__mmask32 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountb_v32qi_mask ((__v32qi) __A,
      (__v32qi)
       _mm256_setzero_si256 (),
      (__mmask32) __U);
}
extern __inline __mmask32
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_bitshuffle_epi64_mask (__m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_vpshufbitqmb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__mmask32) -1);
}
extern __inline __mmask32
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_bitshuffle_epi64_mask (__mmask32 __M, __m256i __A, __m256i __B)
{
  return (__mmask32) __builtin_ia32_vpshufbitqmb256_mask ((__v32qi) __A,
       (__v32qi) __B,
       (__mmask32) __M);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("avx512bitalg,avx512vl")
extern __inline __mmask16
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_bitshuffle_epi64_mask (__m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_vpshufbitqmb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__mmask16) -1);
}
extern __inline __mmask16
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_bitshuffle_epi64_mask (__mmask16 __M, __m128i __A, __m128i __B)
{
  return (__mmask16) __builtin_ia32_vpshufbitqmb128_mask ((__v16qi) __A,
       (__v16qi) __B,
       (__mmask16) __M);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_popcnt_epi8 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountb_v32qi ((__v32qi) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_popcnt_epi16 (__m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountw_v16hi ((__v16hi) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_epi8 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountb_v16qi ((__v16qi) __A);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_popcnt_epi16 (__m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountw_v8hi ((__v8hi) __A);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_popcnt_epi16 (__m256i __A, __mmask16 __U, __m256i __B)
{
  return (__m256i) __builtin_ia32_vpopcountw_v16hi_mask ((__v16hi) __A,
       (__v16hi) __B,
       (__mmask16) __U);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_popcnt_epi16 (__mmask16 __U, __m256i __A)
{
  return (__m256i) __builtin_ia32_vpopcountw_v16hi_mask ((__v16hi) __A,
      (__v16hi)
      _mm256_setzero_si256 (),
      (__mmask16) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_popcnt_epi8 (__m128i __A, __mmask16 __U, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpopcountb_v16qi_mask ((__v16qi) __A,
        (__v16qi) __B,
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_popcnt_epi8 (__mmask16 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountb_v16qi_mask ((__v16qi) __A,
        (__v16qi)
        _mm_setzero_si128 (),
        (__mmask16) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_popcnt_epi16 (__m128i __A, __mmask8 __U, __m128i __B)
{
  return (__m128i) __builtin_ia32_vpopcountw_v8hi_mask ((__v8hi) __A,
       (__v8hi) __B,
       (__mmask8) __U);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_popcnt_epi16 (__mmask8 __U, __m128i __A)
{
  return (__m128i) __builtin_ia32_vpopcountw_v8hi_mask ((__v8hi) __A,
       (__v8hi)
       _mm_setzero_si128 (),
       (__mmask8) __U);
}
#pragma GCC pop_options
# 88 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/shaintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/shaintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("sha")
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha1msg1_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_sha1msg1 ((__v4si) __A, (__v4si) __B);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha1msg2_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_sha1msg2 ((__v4si) __A, (__v4si) __B);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha1nexte_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_sha1nexte ((__v4si) __A, (__v4si) __B);
}
# 71 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/shaintrin.h" 3 4
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha256msg1_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_sha256msg1 ((__v4si) __A, (__v4si) __B);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha256msg2_epu32 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_sha256msg2 ((__v4si) __A, (__v4si) __B);
}
extern __inline __m128i
__attribute__ ((__gnu_inline__, __always_inline__, __artificial__))
_mm_sha256rnds2_epu32 (__m128i __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_sha256rnds2 ((__v4si) __A, (__v4si) __B,
            (__v4si) __C);
}
#pragma GCC pop_options
# 90 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/lzcntintrin.h" 1 3 4
# 33 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/lzcntintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("lzcnt")
extern __inline unsigned short __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__lzcnt16 (unsigned short __X)
{
  return __builtin_ia32_lzcnt_u16 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__lzcnt32 (unsigned int __X)
{
  return __builtin_ia32_lzcnt_u32 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_lzcnt_u32 (unsigned int __X)
{
  return __builtin_ia32_lzcnt_u32 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__lzcnt64 (unsigned long long __X)
{
  return __builtin_ia32_lzcnt_u64 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_lzcnt_u64 (unsigned long long __X)
{
  return __builtin_ia32_lzcnt_u64 (__X);
}
#pragma GCC pop_options
# 92 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/bmiintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/bmiintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("bmi")
extern __inline unsigned short __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__tzcnt_u16 (unsigned short __X)
{
  return __builtin_ia32_tzcnt_u16 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__andn_u32 (unsigned int __X, unsigned int __Y)
{
  return ~__X & __Y;
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__bextr_u32 (unsigned int __X, unsigned int __Y)
{
  return __builtin_ia32_bextr_u32 (__X, __Y);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_bextr_u32 (unsigned int __X, unsigned int __Y, unsigned __Z)
{
  return __builtin_ia32_bextr_u32 (__X, ((__Y & 0xff) | ((__Z & 0xff) << 8)));
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsi_u32 (unsigned int __X)
{
  return __X & -__X;
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsi_u32 (unsigned int __X)
{
  return __blsi_u32 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsmsk_u32 (unsigned int __X)
{
  return __X ^ (__X - 1);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsmsk_u32 (unsigned int __X)
{
  return __blsmsk_u32 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsr_u32 (unsigned int __X)
{
  return __X & (__X - 1);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsr_u32 (unsigned int __X)
{
  return __blsr_u32 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__tzcnt_u32 (unsigned int __X)
{
  return __builtin_ia32_tzcnt_u32 (__X);
}
extern __inline unsigned int __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_tzcnt_u32 (unsigned int __X)
{
  return __builtin_ia32_tzcnt_u32 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__andn_u64 (unsigned long long __X, unsigned long long __Y)
{
  return ~__X & __Y;
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__bextr_u64 (unsigned long long __X, unsigned long long __Y)
{
  return __builtin_ia32_bextr_u64 (__X, __Y);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_bextr_u64 (unsigned long long __X, unsigned int __Y, unsigned int __Z)
{
  return __builtin_ia32_bextr_u64 (__X, ((__Y & 0xff) | ((__Z & 0xff) << 8)));
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsi_u64 (unsigned long long __X)
{
  return __X & -__X;
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsi_u64 (unsigned long long __X)
{
  return __blsi_u64 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsmsk_u64 (unsigned long long __X)
{
  return __X ^ (__X - 1);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsmsk_u64 (unsigned long long __X)
{
  return __blsmsk_u64 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__blsr_u64 (unsigned long long __X)
{
  return __X & (__X - 1);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_blsr_u64 (unsigned long long __X)
{
  return __blsr_u64 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
__tzcnt_u64 (unsigned long long __X)
{
  return __builtin_ia32_tzcnt_u64 (__X);
}
extern __inline unsigned long long __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_tzcnt_u64 (unsigned long long __X)
{
  return __builtin_ia32_tzcnt_u64 (__X);
}
#pragma GCC pop_options
# 94 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/bmi2intrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/bmi2intrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("bmi2")
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_bzhi_u32 (unsigned int __X, unsigned int __Y)
{
  return __builtin_ia32_bzhi_si (__X, __Y);
}
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_pdep_u32 (unsigned int __X, unsigned int __Y)
{
  return __builtin_ia32_pdep_si (__X, __Y);
}
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_pext_u32 (unsigned int __X, unsigned int __Y)
{
  return __builtin_ia32_pext_si (__X, __Y);
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_bzhi_u64 (unsigned long long __X, unsigned long long __Y)
{
  return __builtin_ia32_bzhi_di (__X, __Y);
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_pdep_u64 (unsigned long long __X, unsigned long long __Y)
{
  return __builtin_ia32_pdep_di (__X, __Y);
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_pext_u64 (unsigned long long __X, unsigned long long __Y)
{
  return __builtin_ia32_pext_di (__X, __Y);
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mulx_u64 (unsigned long long __X, unsigned long long __Y,
    unsigned long long *__P)
{
  unsigned __int128 __res = (unsigned __int128) __X * __Y;
  *__P = (unsigned long long) (__res >> 64);
  return (unsigned long long) __res;
}
# 106 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/bmi2intrin.h" 3 4
#pragma GCC pop_options
# 96 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/fmaintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/fmaintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("fma")
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmadd_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddpd ((__v2df)__A, (__v2df)__B,
                                           (__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmadd_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddpd256 ((__v4df)__A, (__v4df)__B,
                                              (__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmadd_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddps ((__v4sf)__A, (__v4sf)__B,
                                          (__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddps256 ((__v8sf)__A, (__v8sf)__B,
                                             (__v8sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmadd_sd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d) __builtin_ia32_vfmaddsd3 ((__v2df)__A, (__v2df)__B,
                                             (__v2df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmadd_ss (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128) __builtin_ia32_vfmaddss3 ((__v4sf)__A, (__v4sf)__B,
                                            (__v4sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsub_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddpd ((__v2df)__A, (__v2df)__B,
                                           -(__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmsub_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddpd256 ((__v4df)__A, (__v4df)__B,
                                              -(__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsub_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddps ((__v4sf)__A, (__v4sf)__B,
                                          -(__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmsub_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddps256 ((__v8sf)__A, (__v8sf)__B,
                                             -(__v8sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsub_sd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddsd3 ((__v2df)__A, (__v2df)__B,
                                            -(__v2df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsub_ss (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddss3 ((__v4sf)__A, (__v4sf)__B,
                                           -(__v4sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmadd_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddpd (-(__v2df)__A, (__v2df)__B,
                                           (__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fnmadd_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddpd256 (-(__v4df)__A, (__v4df)__B,
                                              (__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmadd_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddps (-(__v4sf)__A, (__v4sf)__B,
                                          (__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fnmadd_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddps256 (-(__v8sf)__A, (__v8sf)__B,
                                             (__v8sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmadd_sd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddsd3 ((__v2df)__A, -(__v2df)__B,
                                            (__v2df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmadd_ss (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddss3 ((__v4sf)__A, -(__v4sf)__B,
                                           (__v4sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmsub_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddpd (-(__v2df)__A, (__v2df)__B,
                                           -(__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fnmsub_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddpd256 (-(__v4df)__A, (__v4df)__B,
                                              -(__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmsub_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddps (-(__v4sf)__A, (__v4sf)__B,
                                          -(__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fnmsub_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddps256 (-(__v8sf)__A, (__v8sf)__B,
                                             -(__v8sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmsub_sd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddsd3 ((__v2df)__A, -(__v2df)__B,
                                            -(__v2df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fnmsub_ss (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddss3 ((__v4sf)__A, -(__v4sf)__B,
                                           -(__v4sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmaddsub_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddsubpd ((__v2df)__A, (__v2df)__B,
                                              (__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmaddsub_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddsubpd256 ((__v4df)__A,
                                                 (__v4df)__B,
                                                 (__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmaddsub_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddsubps ((__v4sf)__A, (__v4sf)__B,
                                             (__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmaddsub_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddsubps256 ((__v8sf)__A,
                                                (__v8sf)__B,
                                                (__v8sf)__C);
}
extern __inline __m128d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsubadd_pd (__m128d __A, __m128d __B, __m128d __C)
{
  return (__m128d)__builtin_ia32_vfmaddsubpd ((__v2df)__A, (__v2df)__B,
                                              -(__v2df)__C);
}
extern __inline __m256d
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmsubadd_pd (__m256d __A, __m256d __B, __m256d __C)
{
  return (__m256d)__builtin_ia32_vfmaddsubpd256 ((__v4df)__A,
                                                 (__v4df)__B,
                                                 -(__v4df)__C);
}
extern __inline __m128
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_fmsubadd_ps (__m128 __A, __m128 __B, __m128 __C)
{
  return (__m128)__builtin_ia32_vfmaddsubps ((__v4sf)__A, (__v4sf)__B,
                                             -(__v4sf)__C);
}
extern __inline __m256
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_fmsubadd_ps (__m256 __A, __m256 __B, __m256 __C)
{
  return (__m256)__builtin_ia32_vfmaddsubps256 ((__v8sf)__A,
                                                (__v8sf)__B,
                                                -(__v8sf)__C);
}
#pragma GCC pop_options
# 98 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/f16cintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/f16cintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("f16c")
extern __inline float __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_cvtsh_ss (unsigned short __S)
{
  __v8hi __H = __extension__ (__v8hi){ (short) __S, 0, 0, 0, 0, 0, 0, 0 };
  __v4sf __A = __builtin_ia32_vcvtph2ps (__H);
  return __builtin_ia32_vec_ext_v4sf (__A, 0);
}
extern __inline __m128 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_cvtph_ps (__m128i __A)
{
  return (__m128) __builtin_ia32_vcvtph2ps ((__v8hi) __A);
}
extern __inline __m256 __attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_cvtph_ps (__m128i __A)
{
  return (__m256) __builtin_ia32_vcvtph2ps256 ((__v8hi) __A);
}
# 95 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/f16cintrin.h" 3 4
#pragma GCC pop_options
# 100 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/rtmintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/rtmintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("rtm")
# 48 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/rtmintrin.h" 3 4
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_xbegin (void)
{
  return __builtin_ia32_xbegin ();
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_xend (void)
{
  __builtin_ia32_xend ();
}
# 81 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/rtmintrin.h" 3 4
#pragma GCC pop_options
# 102 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xtestintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/xtestintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("rtm")
extern __inline int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_xtest (void)
{
  return __builtin_ia32_xtest ();
}
#pragma GCC pop_options
# 104 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/cetintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/cetintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target ("shstk")
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_get_ssp (void)
{
  return __builtin_ia32_rdsspq ();
}
# 53 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/cetintrin.h" 3 4
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_inc_ssp (unsigned int __B)
{
  __builtin_ia32_incsspq ((unsigned long long) __B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_saveprevssp (void)
{
  __builtin_ia32_saveprevssp ();
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_rstorssp (void *__B)
{
  __builtin_ia32_rstorssp (__B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_wrssd (unsigned int __B, void *__C)
{
  __builtin_ia32_wrssd (__B, __C);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_wrssq (unsigned long long __B, void *__C)
{
  __builtin_ia32_wrssq (__B, __C);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_wrussd (unsigned int __B, void *__C)
{
  __builtin_ia32_wrussd (__B, __C);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_wrussq (unsigned long long __B, void *__C)
{
  __builtin_ia32_wrussq (__B, __C);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_setssbsy (void)
{
  __builtin_ia32_setssbsy ();
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_clrssbsy (void *__B)
{
  __builtin_ia32_clrssbsy (__B);
}
#pragma GCC pop_options
# 106 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("gfni,sse2")
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_gf2p8mul_epi8 (__m128i __A, __m128i __B)
{
  return (__m128i) __builtin_ia32_vgf2p8mulb_v16qi((__v16qi) __A,
         (__v16qi) __B);
}
# 73 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("gfni,avx")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_gf2p8mul_epi8 (__m256i __A, __m256i __B)
{
  return (__m256i) __builtin_ia32_vgf2p8mulb_v32qi ((__v32qi) __A,
          (__v32qi) __B);
}
# 119 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("gfni,avx512vl")
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_mask_gf2p8mul_epi8 (__m128i __A, __mmask16 __B, __m128i __C, __m128i __D)
{
  return (__m128i) __builtin_ia32_vgf2p8mulb_v16qi_mask ((__v16qi) __C,
        (__v16qi) __D,
        (__v16qi)__A, __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_maskz_gf2p8mul_epi8 (__mmask16 __A, __m128i __B, __m128i __C)
{
  return (__m128i) __builtin_ia32_vgf2p8mulb_v16qi_mask ((__v16qi) __B,
   (__v16qi) __C, (__v16qi) _mm_setzero_si128 (), __A);
}
# 207 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("gfni,avx512vl,avx512bw")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_mask_gf2p8mul_epi8 (__m256i __A, __mmask32 __B, __m256i __C,
      __m256i __D)
{
  return (__m256i) __builtin_ia32_vgf2p8mulb_v32qi_mask ((__v32qi) __C,
        (__v32qi) __D,
        (__v32qi)__A, __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_maskz_gf2p8mul_epi8 (__mmask32 __A, __m256i __B, __m256i __C)
{
  return (__m256i) __builtin_ia32_vgf2p8mulb_v32qi_mask ((__v32qi) __B,
   (__v32qi) __C, (__v32qi) _mm256_setzero_si256 (), __A);
}
# 297 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("gfni,avx512f,avx512bw")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_mask_gf2p8mul_epi8 (__m512i __A, __mmask64 __B, __m512i __C,
      __m512i __D)
{
  return (__m512i) __builtin_ia32_vgf2p8mulb_v64qi_mask ((__v64qi) __C,
     (__v64qi) __D, (__v64qi)__A, __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_maskz_gf2p8mul_epi8 (__mmask64 __A, __m512i __B, __m512i __C)
{
  return (__m512i) __builtin_ia32_vgf2p8mulb_v64qi_mask ((__v64qi) __B,
   (__v64qi) __C, (__v64qi) _mm512_setzero_si512 (), __A);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_gf2p8mul_epi8 (__m512i __A, __m512i __B)
{
  return (__m512i) __builtin_ia32_vgf2p8mulb_v64qi ((__v64qi) __A,
          (__v64qi) __B);
}
# 411 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/gfniintrin.h" 3 4
#pragma GCC pop_options
# 108 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vaesintrin.h" 1 3 4
# 28 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vaesintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("vaes,avx")
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_aesdec_epi128 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_vaesdec_v32qi ((__v32qi) __A, (__v32qi) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_aesdeclast_epi128 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_vaesdeclast_v32qi ((__v32qi) __A,
        (__v32qi) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_aesenc_epi128 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_vaesenc_v32qi ((__v32qi) __A, (__v32qi) __B);
}
extern __inline __m256i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm256_aesenclast_epi128 (__m256i __A, __m256i __B)
{
  return (__m256i)__builtin_ia32_vaesenclast_v32qi ((__v32qi) __A,
        (__v32qi) __B);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("vaes,avx512f")
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_aesdec_epi128 (__m512i __A, __m512i __B)
{
  return (__m512i)__builtin_ia32_vaesdec_v64qi ((__v64qi) __A, (__v64qi) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_aesdeclast_epi128 (__m512i __A, __m512i __B)
{
  return (__m512i)__builtin_ia32_vaesdeclast_v64qi ((__v64qi) __A,
          (__v64qi) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_aesenc_epi128 (__m512i __A, __m512i __B)
{
  return (__m512i)__builtin_ia32_vaesenc_v64qi ((__v64qi) __A, (__v64qi) __B);
}
extern __inline __m512i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm512_aesenclast_epi128 (__m512i __A, __m512i __B)
{
  return (__m512i)__builtin_ia32_vaesenclast_v64qi ((__v64qi) __A,
          (__v64qi) __B);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("vaes,avx512vl")
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesdec_epi128 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_vaesdec_v16qi ((__v16qi) __A, (__v16qi) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesdeclast_epi128 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_vaesdeclast_v16qi ((__v16qi) __A,
          (__v16qi) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesenc_epi128 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_vaesenc_v16qi ((__v16qi) __A, (__v16qi) __B);
}
extern __inline __m128i
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_mm_aesenclast_epi128 (__m128i __A, __m128i __B)
{
  return (__m128i)__builtin_ia32_vaesenclast_v16qi ((__v16qi) __A,
          (__v16qi) __B);
}
#pragma GCC pop_options
# 110 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vpclmulqdqintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vpclmulqdqintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target("vpclmulqdq,avx512f")
# 53 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vpclmulqdqintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("vpclmulqdq,avx512vl")
# 78 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vpclmulqdqintrin.h" 3 4
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("vpclmulqdq,avx512vl")
# 103 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/vpclmulqdqintrin.h" 3 4
#pragma GCC pop_options
# 112 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/movdirintrin.h" 1 3 4
# 32 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/movdirintrin.h" 3 4
#pragma GCC push_options
#pragma GCC target ("movdiri")
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_directstoreu_u32 (void * __P, unsigned int __A)
{
  __builtin_ia32_directstoreu_u32 ((unsigned int *)__P, __A);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_directstoreu_u64 (void * __P, unsigned long long __A)
{
  __builtin_ia32_directstoreu_u64 ((unsigned long long *)__P, __A);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target ("movdir64b")
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_movdir64b (void * __P, const void * __Q)
{
  __builtin_ia32_movdir64b (__P, __Q);
}
#pragma GCC pop_options
# 114 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/immintrin.h" 2 3 4
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_wbinvd (void)
{
  __builtin_ia32_wbinvd ();
}
#pragma GCC push_options
#pragma GCC target("rdrnd")
extern __inline int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_rdrand16_step (unsigned short *__P)
{
  return __builtin_ia32_rdrand16_step (__P);
}
extern __inline int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_rdrand32_step (unsigned int *__P)
{
  return __builtin_ia32_rdrand32_step (__P);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("rdpid")
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_rdpid_u32 (void)
{
  return __builtin_ia32_rdpid ();
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("fsgsbase")
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_readfsbase_u32 (void)
{
  return __builtin_ia32_rdfsbase32 ();
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_readfsbase_u64 (void)
{
  return __builtin_ia32_rdfsbase64 ();
}
extern __inline unsigned int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_readgsbase_u32 (void)
{
  return __builtin_ia32_rdgsbase32 ();
}
extern __inline unsigned long long
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_readgsbase_u64 (void)
{
  return __builtin_ia32_rdgsbase64 ();
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_writefsbase_u32 (unsigned int __B)
{
  __builtin_ia32_wrfsbase32 (__B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_writefsbase_u64 (unsigned long long __B)
{
  __builtin_ia32_wrfsbase64 (__B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_writegsbase_u32 (unsigned int __B)
{
  __builtin_ia32_wrgsbase32 (__B);
}
extern __inline void
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_writegsbase_u64 (unsigned long long __B)
{
  __builtin_ia32_wrgsbase64 (__B);
}
#pragma GCC pop_options
#pragma GCC push_options
#pragma GCC target("rdrnd")
extern __inline int
__attribute__((__gnu_inline__, __always_inline__, __artificial__))
_rdrand64_step (unsigned long long *__P)
{
  return __builtin_ia32_rdrand64_step (__P);
}
#pragma GCC pop_options
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_IMMINTRIN_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: ieeefp.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: ieeefp.h: No such file or directory
 #include <ieeefp.h>
          ^~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <ieeefp.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: ieeefp.h: No such file or directory
 #include <ieeefp.h>
          ^~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: ieeefp.h: No such file or directory #include <ieeefp.h>          ^~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: strings.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/strings.h" 1 3 4
# 21 "/usr/include/strings.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/strings.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 24 "/usr/include/strings.h" 2 3 4
extern int bcmp (const void *__s1, const void *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bcopy (const void *__src, void *__dest, size_t __n)
  __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bzero (void *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 68 "/usr/include/strings.h" 3 4
extern char *index (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 96 "/usr/include/strings.h" 3 4
extern char *rindex (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern int ffs (int __i) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int ffsl (long int __l) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
__extension__ extern int ffsll (long long int __ll)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int strcasecmp (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strncasecmp (const char *__s1, const char *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 126 "/usr/include/strings.h" 2 3 4
extern int strcasecmp_l (const char *__s1, const char *__s2, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern int strncasecmp_l (const char *__s1, const char *__s2,
     size_t __n, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 4)));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_STRINGS_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: inttypes.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/inttypes.h" 1 3 4
# 25 "/usr/include/inttypes.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/inttypes.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 1 3 4
# 9 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 26 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 27 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wchar.h" 1 3 4
# 29 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 30 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 35 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 24 "/usr/include/bits/stdint-uintn.h" 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 38 "/usr/include/stdint.h" 2 3 4
typedef __int_least8_t int_least8_t;
typedef __int_least16_t int_least16_t;
typedef __int_least32_t int_least32_t;
typedef __int_least64_t int_least64_t;
typedef __uint_least8_t uint_least8_t;
typedef __uint_least16_t uint_least16_t;
typedef __uint_least32_t uint_least32_t;
typedef __uint_least64_t uint_least64_t;
typedef signed char int_fast8_t;
typedef long int int_fast16_t;
typedef long int int_fast32_t;
typedef long int int_fast64_t;
# 71 "/usr/include/stdint.h" 3 4
typedef unsigned char uint_fast8_t;
typedef unsigned long int uint_fast16_t;
typedef unsigned long int uint_fast32_t;
typedef unsigned long int uint_fast64_t;
# 87 "/usr/include/stdint.h" 3 4
typedef long int intptr_t;
typedef unsigned long int uintptr_t;
# 101 "/usr/include/stdint.h" 3 4
typedef __intmax_t intmax_t;
typedef __uintmax_t uintmax_t;
# 10 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 2 3 4
# 28 "/usr/include/inttypes.h" 2 3 4
typedef int __gwchar_t;
# 266 "/usr/include/inttypes.h" 3 4
typedef struct
  {
    long int quot;
    long int rem;
  } imaxdiv_t;
# 290 "/usr/include/inttypes.h" 3 4
extern intmax_t imaxabs (intmax_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern imaxdiv_t imaxdiv (intmax_t __numer, intmax_t __denom)
      __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern intmax_t strtoimax (const char *__restrict __nptr,
      char **__restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t strtoumax (const char *__restrict __nptr,
       char ** __restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern intmax_t wcstoimax (const __gwchar_t *__restrict __nptr,
      __gwchar_t **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t wcstoumax (const __gwchar_t *__restrict __nptr,
       __gwchar_t ** __restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
# 432 "/usr/include/inttypes.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_INTTYPES_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sched.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sched.h" 1 3 4
# 22 "/usr/include/sched.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 26 "/usr/include/sched.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 30 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 32 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 33 "/usr/include/sched.h" 2 3 4
typedef __pid_t pid_t;
# 1 "/usr/include/bits/sched.h" 1 3 4
# 74 "/usr/include/bits/sched.h" 3 4
# 1 "/usr/include/bits/types/struct_sched_param.h" 1 3 4
# 23 "/usr/include/bits/types/struct_sched_param.h" 3 4
struct sched_param
{
  int sched_priority;
};
# 75 "/usr/include/bits/sched.h" 2 3 4
# 93 "/usr/include/bits/sched.h" 3 4
# 44 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/cpu-set.h" 1 3 4
# 32 "/usr/include/bits/cpu-set.h" 3 4
typedef unsigned long int __cpu_mask;
typedef struct
{
  __cpu_mask __bits[1024 / (8 * sizeof (__cpu_mask))];
} cpu_set_t;
# 115 "/usr/include/bits/cpu-set.h" 3 4
extern int __sched_cpucount (size_t __setsize, const cpu_set_t *__setp)
     __attribute__ ((__nothrow__ , __leaf__));
extern cpu_set_t *__sched_cpualloc (size_t __count) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void __sched_cpufree (cpu_set_t *__set) __attribute__ ((__nothrow__ , __leaf__));
# 45 "/usr/include/sched.h" 2 3 4
extern int sched_setparam (__pid_t __pid, const struct sched_param *__param)
     __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getparam (__pid_t __pid, struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_setscheduler (__pid_t __pid, int __policy,
          const struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getscheduler (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_yield (void) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_max (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_min (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_rr_get_interval (__pid_t __pid, struct timespec *__t) __attribute__ ((__nothrow__ , __leaf__));
# 129 "/usr/include/sched.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SCHED_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: cxxabi.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: cxxabi.h: No such file or directory
 #include <cxxabi.h>
          ^~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <cxxabi.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: cxxabi.h: No such file or directory
 #include <cxxabi.h>
          ^~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: cxxabi.h: No such file or directory #include <cxxabi.h>          ^~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/systeminfo.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory
 #include <sys/systeminfo.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/systeminfo.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory
 #include <sys/systeminfo.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory #include <sys/systeminfo.h>          ^~~~~~~~~~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: dos.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: dos.h: No such file or directory
 #include <dos.h>
          ^~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <dos.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: dos.h: No such file or directory
 #include <dos.h>
          ^~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: dos.h: No such file or directory #include <dos.h>          ^~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: WindowsX.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: WindowsX.h: No such file or directory
 #include <WindowsX.h>
          ^~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <WindowsX.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: WindowsX.h: No such file or directory
 #include <WindowsX.h>
          ^~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: WindowsX.h: No such file or directory #include <WindowsX.h>          ^~~~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/sysinfo.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/sysinfo.h" 1 3 4
# 21 "/usr/include/sys/sysinfo.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/sys/sysinfo.h" 2 3 4
# 1 "/usr/include/linux/kernel.h" 1 3 4
# 1 "/usr/include/linux/sysinfo.h" 1 3 4
# 1 "/usr/include/linux/types.h" 1 3 4
# 1 "/usr/include/asm/types.h" 1 3 4
# 1 "/usr/include/asm-generic/types.h" 1 3 4
# 1 "/usr/include/asm-generic/int-ll64.h" 1 3 4
# 12 "/usr/include/asm-generic/int-ll64.h" 3 4
# 1 "/usr/include/asm/bitsperlong.h" 1 3 4
# 11 "/usr/include/asm/bitsperlong.h" 3 4
# 1 "/usr/include/asm-generic/bitsperlong.h" 1 3 4
# 12 "/usr/include/asm/bitsperlong.h" 2 3 4
# 13 "/usr/include/asm-generic/int-ll64.h" 2 3 4
# 20 "/usr/include/asm-generic/int-ll64.h" 3 4
typedef __signed__ char __s8;
typedef unsigned char __u8;
typedef __signed__ short __s16;
typedef unsigned short __u16;
typedef __signed__ int __s32;
typedef unsigned int __u32;
__extension__ typedef __signed__ long long __s64;
__extension__ typedef unsigned long long __u64;
# 8 "/usr/include/asm-generic/types.h" 2 3 4
# 6 "/usr/include/asm/types.h" 2 3 4
# 6 "/usr/include/linux/types.h" 2 3 4
# 1 "/usr/include/linux/posix_types.h" 1 3 4
# 1 "/usr/include/linux/stddef.h" 1 3 4
# 6 "/usr/include/linux/posix_types.h" 2 3 4
# 25 "/usr/include/linux/posix_types.h" 3 4
typedef struct {
 unsigned long fds_bits[1024 / (8 * sizeof(long))];
} __kernel_fd_set;
typedef void (*__kernel_sighandler_t)(int);
typedef int __kernel_key_t;
typedef int __kernel_mqd_t;
# 1 "/usr/include/asm/posix_types.h" 1 3 4
# 1 "/usr/include/asm/posix_types_64.h" 1 3 4
# 11 "/usr/include/asm/posix_types_64.h" 3 4
typedef unsigned short __kernel_old_uid_t;
typedef unsigned short __kernel_old_gid_t;
typedef unsigned long __kernel_old_dev_t;
# 1 "/usr/include/asm-generic/posix_types.h" 1 3 4
# 15 "/usr/include/asm-generic/posix_types.h" 3 4
typedef long __kernel_long_t;
typedef unsigned long __kernel_ulong_t;
typedef __kernel_ulong_t __kernel_ino_t;
typedef unsigned int __kernel_mode_t;
typedef int __kernel_pid_t;
typedef int __kernel_ipc_pid_t;
typedef unsigned int __kernel_uid_t;
typedef unsigned int __kernel_gid_t;
typedef __kernel_long_t __kernel_suseconds_t;
typedef int __kernel_daddr_t;
typedef unsigned int __kernel_uid32_t;
typedef unsigned int __kernel_gid32_t;
# 72 "/usr/include/asm-generic/posix_types.h" 3 4
typedef __kernel_ulong_t __kernel_size_t;
typedef __kernel_long_t __kernel_ssize_t;
typedef __kernel_long_t __kernel_ptrdiff_t;
typedef struct {
 int val[2];
} __kernel_fsid_t;
typedef __kernel_long_t __kernel_off_t;
typedef long long __kernel_loff_t;
typedef __kernel_long_t __kernel_time_t;
typedef __kernel_long_t __kernel_clock_t;
typedef int __kernel_timer_t;
typedef int __kernel_clockid_t;
typedef char * __kernel_caddr_t;
typedef unsigned short __kernel_uid16_t;
typedef unsigned short __kernel_gid16_t;
# 19 "/usr/include/asm/posix_types_64.h" 2 3 4
# 8 "/usr/include/asm/posix_types.h" 2 3 4
# 37 "/usr/include/linux/posix_types.h" 2 3 4
# 10 "/usr/include/linux/types.h" 2 3 4
# 24 "/usr/include/linux/types.h" 3 4
typedef __u16 __le16;
typedef __u16 __be16;
typedef __u32 __le32;
typedef __u32 __be32;
typedef __u64 __le64;
typedef __u64 __be64;
typedef __u16 __sum16;
typedef __u32 __wsum;
# 50 "/usr/include/linux/types.h" 3 4
typedef unsigned __poll_t;
# 6 "/usr/include/linux/sysinfo.h" 2 3 4
struct sysinfo {
 __kernel_long_t uptime;
 __kernel_ulong_t loads[3];
 __kernel_ulong_t totalram;
 __kernel_ulong_t freeram;
 __kernel_ulong_t sharedram;
 __kernel_ulong_t bufferram;
 __kernel_ulong_t totalswap;
 __kernel_ulong_t freeswap;
 __u16 procs;
 __u16 pad;
 __kernel_ulong_t totalhigh;
 __kernel_ulong_t freehigh;
 __u32 mem_unit;
 char _f[20-2*sizeof(__kernel_ulong_t)-sizeof(__u32)];
};
# 6 "/usr/include/linux/kernel.h" 2 3 4
# 25 "/usr/include/sys/sysinfo.h" 2 3 4
extern int sysinfo (struct sysinfo *__info) __attribute__ ((__nothrow__ , __leaf__));
extern int get_nprocs_conf (void) __attribute__ ((__nothrow__ , __leaf__));
extern int get_nprocs (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int get_phys_pages (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int get_avphys_pages (void) __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_SYSINFO_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: zmmintrin.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: zmmintrin.h: No such file or directory
 #include <zmmintrin.h>
          ^~~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <zmmintrin.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: zmmintrin.h: No such file or directory
 #include <zmmintrin.h>
          ^~~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: zmmintrin.h: No such file or directory #include <zmmintrin.h>          ^~~~~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/wait.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/wait.h" 1 3 4
# 25 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/wait.h" 2 3 4
typedef __pid_t pid_t;
# 1 "/usr/include/signal.h" 1 3 4
# 27 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/signum.h" 1 3 4
# 26 "/usr/include/bits/signum.h" 3 4
# 1 "/usr/include/bits/signum-generic.h" 1 3 4
# 27 "/usr/include/bits/signum.h" 2 3 4
# 31 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sig_atomic_t.h" 1 3 4
typedef __sig_atomic_t sig_atomic_t;
# 33 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 36 "/usr/include/signal.h" 2 3 4
# 46 "/usr/include/signal.h" 3 4
typedef __uid_t uid_t;
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 54 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/siginfo_t.h" 1 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 5 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 1 "/usr/include/bits/types/__sigval_t.h" 1 3 4
# 24 "/usr/include/bits/types/__sigval_t.h" 3 4
union sigval
{
  int sival_int;
  void *sival_ptr;
};
typedef union sigval __sigval_t;
# 7 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 16 "/usr/include/bits/types/siginfo_t.h" 3 4
# 1 "/usr/include/bits/siginfo-arch.h" 1 3 4
# 17 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 36 "/usr/include/bits/types/siginfo_t.h" 3 4
typedef struct
  {
    int si_signo;
    int si_errno;
    int si_code;
    int __pad0;
    union
      {
 int _pad[((128 / sizeof (int)) - 4)];
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
   } _kill;
 struct
   {
     int si_tid;
     int si_overrun;
     __sigval_t si_sigval;
   } _timer;
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
     __sigval_t si_sigval;
   } _rt;
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
     int si_status;
     __clock_t si_utime;
     __clock_t si_stime;
   } _sigchld;
 struct
   {
     void *si_addr;
    
     short int si_addr_lsb;
     union
       {
  struct
    {
      void *_lower;
      void *_upper;
    } _addr_bnd;
  __uint32_t _pkey;
       } _bounds;
   } _sigfault;
 struct
   {
     long int si_band;
     int si_fd;
   } _sigpoll;
 struct
   {
     void *_call_addr;
     int _syscall;
     unsigned int _arch;
   } _sigsys;
      } _sifields;
  } siginfo_t ;
# 58 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/siginfo-consts.h" 1 3 4
# 35 "/usr/include/bits/siginfo-consts.h" 3 4
enum
{
  SI_ASYNCNL = -60,
  SI_TKILL = -6,
  SI_SIGIO,
  SI_ASYNCIO,
  SI_MESGQ,
  SI_TIMER,
  SI_QUEUE,
  SI_USER,
  SI_KERNEL = 0x80
# 63 "/usr/include/bits/siginfo-consts.h" 3 4
};
enum
{
  ILL_ILLOPC = 1,
  ILL_ILLOPN,
  ILL_ILLADR,
  ILL_ILLTRP,
  ILL_PRVOPC,
  ILL_PRVREG,
  ILL_COPROC,
  ILL_BADSTK
};
enum
{
  FPE_INTDIV = 1,
  FPE_INTOVF,
  FPE_FLTDIV,
  FPE_FLTOVF,
  FPE_FLTUND,
  FPE_FLTRES,
  FPE_FLTINV,
  FPE_FLTSUB
};
enum
{
  SEGV_MAPERR = 1,
  SEGV_ACCERR,
  SEGV_BNDERR,
  SEGV_PKUERR
};
enum
{
  BUS_ADRALN = 1,
  BUS_ADRERR,
  BUS_OBJERR,
  BUS_MCEERR_AR,
  BUS_MCEERR_AO
};
# 151 "/usr/include/bits/siginfo-consts.h" 3 4
enum
{
  CLD_EXITED = 1,
  CLD_KILLED,
  CLD_DUMPED,
  CLD_TRAPPED,
  CLD_STOPPED,
  CLD_CONTINUED
};
enum
{
  POLL_IN = 1,
  POLL_OUT,
  POLL_MSG,
  POLL_ERR,
  POLL_PRI,
  POLL_HUP
};
# 59 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sigval_t.h" 1 3 4
# 16 "/usr/include/bits/types/sigval_t.h" 3 4
typedef __sigval_t sigval_t;
# 63 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sigevent_t.h" 1 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 5 "/usr/include/bits/types/sigevent_t.h" 2 3 4
# 17 "/usr/include/bits/types/sigevent_t.h" 3 4
typedef union pthread_attr_t pthread_attr_t;
typedef struct sigevent
  {
    __sigval_t sigev_value;
    int sigev_signo;
    int sigev_notify;
    union
      {
 int _pad[((64 / sizeof (int)) - 4)];
 __pid_t _tid;
 struct
   {
     void (*_function) (__sigval_t);
     pthread_attr_t *_attribute;
   } _sigev_thread;
      } _sigev_un;
  } sigevent_t;
# 67 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/sigevent-consts.h" 1 3 4
# 27 "/usr/include/bits/sigevent-consts.h" 3 4
enum
{
  SIGEV_SIGNAL = 0,
  SIGEV_NONE,
  SIGEV_THREAD,
  SIGEV_THREAD_ID = 4
};
# 68 "/usr/include/signal.h" 2 3 4
typedef void (*__sighandler_t) (int);
extern __sighandler_t __sysv_signal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
# 88 "/usr/include/signal.h" 3 4
extern __sighandler_t signal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
# 112 "/usr/include/signal.h" 3 4
extern int kill (__pid_t __pid, int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern int killpg (__pid_t __pgrp, int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern int raise (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern __sighandler_t ssignal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
extern int gsignal (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern void psignal (int __sig, const char *__s);
extern void psiginfo (const siginfo_t *__pinfo, const char *__s);
# 170 "/usr/include/signal.h" 3 4
extern int sigblock (int __mask) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
extern int sigsetmask (int __mask) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
extern int siggetmask (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
# 190 "/usr/include/signal.h" 3 4
typedef __sighandler_t sig_t;
extern int sigemptyset (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigfillset (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigaddset (sigset_t *__set, int __signo) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigdelset (sigset_t *__set, int __signo) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigismember (const sigset_t *__set, int __signo)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 226 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/sigaction.h" 1 3 4
# 27 "/usr/include/bits/sigaction.h" 3 4
struct sigaction
  {
    union
      {
 __sighandler_t sa_handler;
 void (*sa_sigaction) (int, siginfo_t *, void *);
      }
    __sigaction_handler;
    __sigset_t sa_mask;
    int sa_flags;
    void (*sa_restorer) (void);
  };
# 227 "/usr/include/signal.h" 2 3 4
extern int sigprocmask (int __how, const sigset_t *__restrict __set,
   sigset_t *__restrict __oset) __attribute__ ((__nothrow__ , __leaf__));
extern int sigsuspend (const sigset_t *__set) __attribute__ ((__nonnull__ (1)));
extern int sigaction (int __sig, const struct sigaction *__restrict __act,
        struct sigaction *__restrict __oact) __attribute__ ((__nothrow__ , __leaf__));
extern int sigpending (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigwait (const sigset_t *__restrict __set, int *__restrict __sig)
     __attribute__ ((__nonnull__ (1, 2)));
extern int sigwaitinfo (const sigset_t *__restrict __set,
   siginfo_t *__restrict __info) __attribute__ ((__nonnull__ (1)));
extern int sigtimedwait (const sigset_t *__restrict __set,
    siginfo_t *__restrict __info,
    const struct timespec *__restrict __timeout)
     __attribute__ ((__nonnull__ (1)));
extern int sigqueue (__pid_t __pid, int __sig, const union sigval __val)
     __attribute__ ((__nothrow__ , __leaf__));
# 286 "/usr/include/signal.h" 3 4
extern const char *const _sys_siglist[(64 + 1)];
extern const char *const sys_siglist[(64 + 1)];
# 1 "/usr/include/bits/sigcontext.h" 1 3 4
# 31 "/usr/include/bits/sigcontext.h" 3 4
struct _fpx_sw_bytes
{
  __uint32_t magic1;
  __uint32_t extended_size;
  __uint64_t xstate_bv;
  __uint32_t xstate_size;
  __uint32_t __glibc_reserved1[7];
};
struct _fpreg
{
  unsigned short significand[4];
  unsigned short exponent;
};
struct _fpxreg
{
  unsigned short significand[4];
  unsigned short exponent;
  unsigned short __glibc_reserved1[3];
};
struct _xmmreg
{
  __uint32_t element[4];
};
# 123 "/usr/include/bits/sigcontext.h" 3 4
struct _fpstate
{
  __uint16_t cwd;
  __uint16_t swd;
  __uint16_t ftw;
  __uint16_t fop;
  __uint64_t rip;
  __uint64_t rdp;
  __uint32_t mxcsr;
  __uint32_t mxcr_mask;
  struct _fpxreg _st[8];
  struct _xmmreg _xmm[16];
  __uint32_t __glibc_reserved1[24];
};
struct sigcontext
{
  __uint64_t r8;
  __uint64_t r9;
  __uint64_t r10;
  __uint64_t r11;
  __uint64_t r12;
  __uint64_t r13;
  __uint64_t r14;
  __uint64_t r15;
  __uint64_t rdi;
  __uint64_t rsi;
  __uint64_t rbp;
  __uint64_t rbx;
  __uint64_t rdx;
  __uint64_t rax;
  __uint64_t rcx;
  __uint64_t rsp;
  __uint64_t rip;
  __uint64_t eflags;
  unsigned short cs;
  unsigned short gs;
  unsigned short fs;
  unsigned short __pad0;
  __uint64_t err;
  __uint64_t trapno;
  __uint64_t oldmask;
  __uint64_t cr2;
  __extension__ union
    {
      struct _fpstate * fpstate;
      __uint64_t __fpstate_word;
    };
  __uint64_t __reserved1 [8];
};
struct _xsave_hdr
{
  __uint64_t xstate_bv;
  __uint64_t __glibc_reserved1[2];
  __uint64_t __glibc_reserved2[5];
};
struct _ymmh_state
{
  __uint32_t ymmh_space[64];
};
struct _xstate
{
  struct _fpstate fpstate;
  struct _xsave_hdr xstate_hdr;
  struct _ymmh_state ymmh;
};
# 292 "/usr/include/signal.h" 2 3 4
extern int sigreturn (struct sigcontext *__scp) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 302 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/stack_t.h" 1 3 4
# 23 "/usr/include/bits/types/stack_t.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 24 "/usr/include/bits/types/stack_t.h" 2 3 4
typedef struct
  {
    void *ss_sp;
    int ss_flags;
    size_t ss_size;
  } stack_t;
# 304 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/sys/ucontext.h" 1 3 4
# 37 "/usr/include/sys/ucontext.h" 3 4
__extension__ typedef long long int greg_t;
# 46 "/usr/include/sys/ucontext.h" 3 4
typedef greg_t gregset_t[23];
# 101 "/usr/include/sys/ucontext.h" 3 4
struct _libc_fpxreg
{
  unsigned short int significand[4];
  unsigned short int exponent;
  unsigned short int __glibc_reserved1[3];
};
struct _libc_xmmreg
{
  __uint32_t element[4];
};
struct _libc_fpstate
{
  __uint16_t cwd;
  __uint16_t swd;
  __uint16_t ftw;
  __uint16_t fop;
  __uint64_t rip;
  __uint64_t rdp;
  __uint32_t mxcsr;
  __uint32_t mxcr_mask;
  struct _libc_fpxreg _st[8];
  struct _libc_xmmreg _xmm[16];
  __uint32_t __glibc_reserved1[24];
};
typedef struct _libc_fpstate *fpregset_t;
typedef struct
  {
    gregset_t gregs;
    fpregset_t fpregs;
    __extension__ unsigned long long __reserved1 [8];
} mcontext_t;
typedef struct ucontext_t
  {
    unsigned long int uc_flags;
    struct ucontext_t *uc_link;
    stack_t uc_stack;
    mcontext_t uc_mcontext;
    sigset_t uc_sigmask;
    struct _libc_fpstate __fpregs_mem;
    __extension__ unsigned long long int __ssp[4];
  } ucontext_t;
# 307 "/usr/include/signal.h" 2 3 4
extern int siginterrupt (int __sig, int __interrupt) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/bits/sigstack.h" 1 3 4
# 317 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/ss_flags.h" 1 3 4
# 27 "/usr/include/bits/ss_flags.h" 3 4
enum
{
  SS_ONSTACK = 1,
  SS_DISABLE
};
# 318 "/usr/include/signal.h" 2 3 4
extern int sigaltstack (const stack_t *__restrict __ss,
   stack_t *__restrict __oss) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/bits/types/struct_sigstack.h" 1 3 4
# 23 "/usr/include/bits/types/struct_sigstack.h" 3 4
struct sigstack
  {
    void *ss_sp;
    int ss_onstack;
  };
# 328 "/usr/include/signal.h" 2 3 4
extern int sigstack (struct sigstack *__ss, struct sigstack *__oss)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
# 358 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 359 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/sigthread.h" 1 3 4
# 31 "/usr/include/bits/sigthread.h" 3 4
extern int pthread_sigmask (int __how,
       const __sigset_t *__restrict __newmask,
       __sigset_t *__restrict __oldmask)__attribute__ ((__nothrow__ , __leaf__));
extern int pthread_kill (pthread_t __threadid, int __signo) __attribute__ ((__nothrow__ , __leaf__));
# 360 "/usr/include/signal.h" 2 3 4
extern int __libc_current_sigrtmin (void) __attribute__ ((__nothrow__ , __leaf__));
extern int __libc_current_sigrtmax (void) __attribute__ ((__nothrow__ , __leaf__));
# 37 "/usr/include/sys/wait.h" 2 3 4
# 49 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 50 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 53 "/usr/include/sys/wait.h" 2 3 4
# 74 "/usr/include/sys/wait.h" 3 4
typedef enum
{
  P_ALL,
  P_PID,
  P_PGID
} idtype_t;
# 88 "/usr/include/sys/wait.h" 3 4
extern __pid_t wait (int *__stat_loc);
# 111 "/usr/include/sys/wait.h" 3 4
extern __pid_t waitpid (__pid_t __pid, int *__stat_loc, int __options);
typedef __id_t id_t;
# 132 "/usr/include/sys/wait.h" 3 4
extern int waitid (idtype_t __idtype, __id_t __id, siginfo_t *__infop,
     int __options);
struct rusage;
extern __pid_t wait3 (int *__stat_loc, int __options,
        struct rusage * __usage) __attribute__ ((__nothrow__));
extern __pid_t wait4 (__pid_t __pid, int *__stat_loc, int __options,
        struct rusage *__usage) __attribute__ ((__nothrow__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_WAIT_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: stdlib.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/stdlib.h" 1 3 4
# 25 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 26 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 32 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitflags.h" 1 3 4
# 40 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/bits/waitstatus.h" 1 3 4
# 41 "/usr/include/stdlib.h" 2 3 4
# 55 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/floatn.h" 1 3 4
# 119 "/usr/include/bits/floatn.h" 3 4
# 1 "/usr/include/bits/floatn-common.h" 1 3 4
# 24 "/usr/include/bits/floatn-common.h" 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 25 "/usr/include/bits/floatn-common.h" 2 3 4
# 120 "/usr/include/bits/floatn.h" 2 3 4
# 56 "/usr/include/stdlib.h" 2 3 4
typedef struct
  {
    int quot;
    int rem;
  } div_t;
typedef struct
  {
    long int quot;
    long int rem;
  } ldiv_t;
__extension__ typedef struct
  {
    long long int quot;
    long long int rem;
  } lldiv_t;
# 97 "/usr/include/stdlib.h" 3 4
extern size_t __ctype_get_mb_cur_max (void) __attribute__ ((__nothrow__ , __leaf__)) ;
extern double atof (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern int atoi (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern long int atol (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
__extension__ extern long long int atoll (const char *__nptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
extern double strtod (const char *__restrict __nptr,
        char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern float strtof (const char *__restrict __nptr,
       char **__restrict __endptr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long double strtold (const char *__restrict __nptr,
       char **__restrict __endptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 176 "/usr/include/stdlib.h" 3 4
extern long int strtol (const char *__restrict __nptr,
   char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern unsigned long int strtoul (const char *__restrict __nptr,
      char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoq (const char *__restrict __nptr,
        char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtouq (const char *__restrict __nptr,
           char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern long long int strtoll (const char *__restrict __nptr,
         char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
__extension__
extern unsigned long long int strtoull (const char *__restrict __nptr,
     char **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 385 "/usr/include/stdlib.h" 3 4
extern char *l64a (long int __n) __attribute__ ((__nothrow__ , __leaf__)) ;
extern long int a64l (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1))) ;
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 395 "/usr/include/stdlib.h" 2 3 4
extern long int random (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srandom (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern char *initstate (unsigned int __seed, char *__statebuf,
   size_t __statelen) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *setstate (char *__statebuf) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct random_data
  {
    int32_t *fptr;
    int32_t *rptr;
    int32_t *state;
    int rand_type;
    int rand_deg;
    int rand_sep;
    int32_t *end_ptr;
  };
extern int random_r (struct random_data *__restrict __buf,
       int32_t *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srandom_r (unsigned int __seed, struct random_data *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int initstate_r (unsigned int __seed, char *__restrict __statebuf,
   size_t __statelen,
   struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern int setstate_r (char *__restrict __statebuf,
         struct random_data *__restrict __buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int rand (void) __attribute__ ((__nothrow__ , __leaf__));
extern void srand (unsigned int __seed) __attribute__ ((__nothrow__ , __leaf__));
extern int rand_r (unsigned int *__seed) __attribute__ ((__nothrow__ , __leaf__));
extern double drand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern double erand48 (unsigned short int __xsubi[3]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int lrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int nrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int mrand48 (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int jrand48 (unsigned short int __xsubi[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void srand48 (long int __seedval) __attribute__ ((__nothrow__ , __leaf__));
extern unsigned short int *seed48 (unsigned short int __seed16v[3])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void lcong48 (unsigned short int __param[7]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
struct drand48_data
  {
    unsigned short int __x[3];
    unsigned short int __old_x[3];
    unsigned short int __c;
    unsigned short int __init;
    __extension__ unsigned long long int __a;
  };
extern int drand48_r (struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int erand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        double *__restrict __result) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int nrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int mrand48_r (struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int jrand48_r (unsigned short int __xsubi[3],
        struct drand48_data *__restrict __buffer,
        long int *__restrict __result)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int srand48_r (long int __seedval, struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int seed48_r (unsigned short int __seed16v[3],
       struct drand48_data *__buffer) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int lcong48_r (unsigned short int __param[7],
        struct drand48_data *__buffer)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *malloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *calloc (size_t __nmemb, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern void *realloc (void *__ptr, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__warn_unused_result__));
# 563 "/usr/include/stdlib.h" 3 4
extern void free (void *__ptr) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/alloca.h" 1 3 4
# 24 "/usr/include/alloca.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 25 "/usr/include/alloca.h" 2 3 4
extern void *alloca (size_t __size) __attribute__ ((__nothrow__ , __leaf__));
# 567 "/usr/include/stdlib.h" 2 3 4
extern void *valloc (size_t __size) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) ;
extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern void *aligned_alloc (size_t __alignment, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__alloc_size__ (2))) ;
extern void abort (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern int atexit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int at_quick_exit (void (*__func) (void)) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int on_exit (void (*__func) (int __status, void *__arg), void *__arg)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern void exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void quick_exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern void _Exit (int __status) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__noreturn__));
extern char *getenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 644 "/usr/include/stdlib.h" 3 4
extern int putenv (char *__string) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int setenv (const char *__name, const char *__value, int __replace)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int unsetenv (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int clearenv (void) __attribute__ ((__nothrow__ , __leaf__));
# 672 "/usr/include/stdlib.h" 3 4
extern char *mktemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 685 "/usr/include/stdlib.h" 3 4
extern int mkstemp (char *__template) __attribute__ ((__nonnull__ (1))) ;
# 707 "/usr/include/stdlib.h" 3 4
extern int mkstemps (char *__template, int __suffixlen) __attribute__ ((__nonnull__ (1))) ;
# 728 "/usr/include/stdlib.h" 3 4
extern char *mkdtemp (char *__template) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 781 "/usr/include/stdlib.h" 3 4
extern int system (const char *__command) ;
# 797 "/usr/include/stdlib.h" 3 4
extern char *realpath (const char *__restrict __name,
         char *__restrict __resolved) __attribute__ ((__nothrow__ , __leaf__)) ;
typedef int (*__compar_fn_t) (const void *, const void *);
# 817 "/usr/include/stdlib.h" 3 4
extern void *bsearch (const void *__key, const void *__base,
        size_t __nmemb, size_t __size, __compar_fn_t __compar)
     __attribute__ ((__nonnull__ (1, 2, 5))) ;
extern void qsort (void *__base, size_t __nmemb, size_t __size,
     __compar_fn_t __compar) __attribute__ ((__nonnull__ (1, 4)));
# 837 "/usr/include/stdlib.h" 3 4
extern int abs (int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern long int labs (long int __x) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern long long int llabs (long long int __x)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern div_t div (int __numer, int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
extern ldiv_t ldiv (long int __numer, long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
__extension__ extern lldiv_t lldiv (long long int __numer,
        long long int __denom)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__)) ;
# 869 "/usr/include/stdlib.h" 3 4
extern char *ecvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *fcvt (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *gcvt (double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern char *qecvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qfcvt (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4))) ;
extern char *qgcvt (long double __value, int __ndigit, char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3))) ;
extern int ecvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int fcvt_r (double __value, int __ndigit, int *__restrict __decpt,
     int *__restrict __sign, char *__restrict __buf,
     size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qecvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int qfcvt_r (long double __value, int __ndigit,
      int *__restrict __decpt, int *__restrict __sign,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3, 4, 5)));
extern int mblen (const char *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int mbtowc (wchar_t *__restrict __pwc,
     const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern int wctomb (char *__s, wchar_t __wchar) __attribute__ ((__nothrow__ , __leaf__));
extern size_t mbstowcs (wchar_t *__restrict __pwcs,
   const char *__restrict __s, size_t __n) __attribute__ ((__nothrow__ , __leaf__));
extern size_t wcstombs (char *__restrict __s,
   const wchar_t *__restrict __pwcs, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__));
extern int rpmatch (const char *__response) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 954 "/usr/include/stdlib.h" 3 4
extern int getsubopt (char **__restrict __optionp,
        char *const *__restrict __tokens,
        char **__restrict __valuep)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3))) ;
# 1000 "/usr/include/stdlib.h" 3 4
extern int getloadavg (double __loadavg[], int __nelem)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1010 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/bits/stdlib-float.h" 1 3 4
# 1011 "/usr/include/stdlib.h" 2 3 4
# 1020 "/usr/include/stdlib.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_STDLIB_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: pthread.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/pthread.h" 1 3 4
# 21 "/usr/include/pthread.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 27 "/usr/include/bits/byteswap.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/bits/byteswap.h" 2 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 23 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/sched.h" 1 3 4
# 29 "/usr/include/sched.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 30 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 32 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 33 "/usr/include/sched.h" 2 3 4
typedef __pid_t pid_t;
# 1 "/usr/include/bits/sched.h" 1 3 4
# 74 "/usr/include/bits/sched.h" 3 4
# 1 "/usr/include/bits/types/struct_sched_param.h" 1 3 4
# 23 "/usr/include/bits/types/struct_sched_param.h" 3 4
struct sched_param
{
  int sched_priority;
};
# 75 "/usr/include/bits/sched.h" 2 3 4
# 93 "/usr/include/bits/sched.h" 3 4
# 44 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/cpu-set.h" 1 3 4
# 32 "/usr/include/bits/cpu-set.h" 3 4
typedef unsigned long int __cpu_mask;
typedef struct
{
  __cpu_mask __bits[1024 / (8 * sizeof (__cpu_mask))];
} cpu_set_t;
# 115 "/usr/include/bits/cpu-set.h" 3 4
extern int __sched_cpucount (size_t __setsize, const cpu_set_t *__setp)
     __attribute__ ((__nothrow__ , __leaf__));
extern cpu_set_t *__sched_cpualloc (size_t __count) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void __sched_cpufree (cpu_set_t *__set) __attribute__ ((__nothrow__ , __leaf__));
# 45 "/usr/include/sched.h" 2 3 4
extern int sched_setparam (__pid_t __pid, const struct sched_param *__param)
     __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getparam (__pid_t __pid, struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_setscheduler (__pid_t __pid, int __policy,
          const struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getscheduler (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_yield (void) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_max (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_min (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_rr_get_interval (__pid_t __pid, struct timespec *__t) __attribute__ ((__nothrow__ , __leaf__));
# 129 "/usr/include/sched.h" 3 4
# 24 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/time.h" 1 3 4
# 29 "/usr/include/time.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 30 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/time.h" 1 3 4
# 34 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 38 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_tm.h" 1 3 4
struct tm
{
  int tm_sec;
  int tm_min;
  int tm_hour;
  int tm_mday;
  int tm_mon;
  int tm_year;
  int tm_wday;
  int tm_yday;
  int tm_isdst;
  long int tm_gmtoff;
  const char *tm_zone;
};
# 40 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 47 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 48 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_itimerspec.h" 1 3 4
struct itimerspec
  {
    struct timespec it_interval;
    struct timespec it_value;
  };
# 49 "/usr/include/time.h" 2 3 4
struct sigevent;
# 60 "/usr/include/time.h" 3 4
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 61 "/usr/include/time.h" 2 3 4
extern clock_t clock (void) __attribute__ ((__nothrow__ , __leaf__));
extern time_t time (time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern double difftime (time_t __time1, time_t __time0)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern time_t mktime (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern size_t strftime (char *__restrict __s, size_t __maxsize,
   const char *__restrict __format,
   const struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
# 104 "/usr/include/time.h" 3 4
extern size_t strftime_l (char *__restrict __s, size_t __maxsize,
     const char *__restrict __format,
     const struct tm *__restrict __tp,
     locale_t __loc) __attribute__ ((__nothrow__ , __leaf__));
# 119 "/usr/include/time.h" 3 4
extern struct tm *gmtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *gmtime_r (const time_t *__restrict __timer,
       struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime_r (const time_t *__restrict __timer,
          struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime (const struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime_r (const struct tm *__restrict __tp,
   char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime_r (const time_t *__restrict __timer,
        char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *__tzname[2];
extern int __daylight;
extern long int __timezone;
extern char *tzname[2];
extern void tzset (void) __attribute__ ((__nothrow__ , __leaf__));
extern int daylight;
extern long int timezone;
extern int stime (const time_t *__when) __attribute__ ((__nothrow__ , __leaf__));
# 196 "/usr/include/time.h" 3 4
extern time_t timegm (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern time_t timelocal (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int dysize (int __year) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 211 "/usr/include/time.h" 3 4
extern int nanosleep (const struct timespec *__requested_time,
        struct timespec *__remaining);
extern int clock_getres (clockid_t __clock_id, struct timespec *__res) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_gettime (clockid_t __clock_id, struct timespec *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_settime (clockid_t __clock_id, const struct timespec *__tp)
     __attribute__ ((__nothrow__ , __leaf__));
extern int clock_nanosleep (clockid_t __clock_id, int __flags,
       const struct timespec *__req,
       struct timespec *__rem);
extern int clock_getcpuclockid (pid_t __pid, clockid_t *__clock_id) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_create (clockid_t __clock_id,
    struct sigevent *__restrict __evp,
    timer_t *__restrict __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_delete (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_settime (timer_t __timerid, int __flags,
     const struct itimerspec *__restrict __value,
     struct itimerspec *__restrict __ovalue) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_gettime (timer_t __timerid, struct itimerspec *__value)
     __attribute__ ((__nothrow__ , __leaf__));
extern int timer_getoverrun (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timespec_get (struct timespec *__ts, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 307 "/usr/include/time.h" 3 4
# 25 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 27 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/setjmp.h" 1 3 4
# 26 "/usr/include/bits/setjmp.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 27 "/usr/include/bits/setjmp.h" 2 3 4
typedef long int __jmp_buf[8];
# 28 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 29 "/usr/include/pthread.h" 2 3 4
enum
{
  PTHREAD_CREATE_JOINABLE,
  PTHREAD_CREATE_DETACHED
};
enum
{
  PTHREAD_MUTEX_TIMED_NP,
  PTHREAD_MUTEX_RECURSIVE_NP,
  PTHREAD_MUTEX_ERRORCHECK_NP,
  PTHREAD_MUTEX_ADAPTIVE_NP
  ,
  PTHREAD_MUTEX_NORMAL = PTHREAD_MUTEX_TIMED_NP,
  PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP,
  PTHREAD_MUTEX_ERRORCHECK = PTHREAD_MUTEX_ERRORCHECK_NP,
  PTHREAD_MUTEX_DEFAULT = PTHREAD_MUTEX_NORMAL
};
enum
{
  PTHREAD_MUTEX_STALLED,
  PTHREAD_MUTEX_STALLED_NP = PTHREAD_MUTEX_STALLED,
  PTHREAD_MUTEX_ROBUST,
  PTHREAD_MUTEX_ROBUST_NP = PTHREAD_MUTEX_ROBUST
};
enum
{
  PTHREAD_PRIO_NONE,
  PTHREAD_PRIO_INHERIT,
  PTHREAD_PRIO_PROTECT
};
# 115 "/usr/include/pthread.h" 3 4
enum
{
  PTHREAD_RWLOCK_PREFER_READER_NP,
  PTHREAD_RWLOCK_PREFER_WRITER_NP,
  PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
  PTHREAD_RWLOCK_DEFAULT_NP = PTHREAD_RWLOCK_PREFER_READER_NP
};
# 156 "/usr/include/pthread.h" 3 4
enum
{
  PTHREAD_INHERIT_SCHED,
  PTHREAD_EXPLICIT_SCHED
};
enum
{
  PTHREAD_SCOPE_SYSTEM,
  PTHREAD_SCOPE_PROCESS
};
enum
{
  PTHREAD_PROCESS_PRIVATE,
  PTHREAD_PROCESS_SHARED
};
# 191 "/usr/include/pthread.h" 3 4
struct _pthread_cleanup_buffer
{
  void (*__routine) (void *);
  void *__arg;
  int __canceltype;
  struct _pthread_cleanup_buffer *__prev;
};
enum
{
  PTHREAD_CANCEL_ENABLE,
  PTHREAD_CANCEL_DISABLE
};
enum
{
  PTHREAD_CANCEL_DEFERRED,
  PTHREAD_CANCEL_ASYNCHRONOUS
};
# 229 "/usr/include/pthread.h" 3 4
extern int pthread_create (pthread_t *__restrict __newthread,
      const pthread_attr_t *__restrict __attr,
      void *(*__start_routine) (void *),
      void *__restrict __arg) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 3)));
extern void pthread_exit (void *__retval) __attribute__ ((__noreturn__));
extern int pthread_join (pthread_t __th, void **__thread_return);
# 272 "/usr/include/pthread.h" 3 4
extern int pthread_detach (pthread_t __th) __attribute__ ((__nothrow__ , __leaf__));
extern pthread_t pthread_self (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int pthread_equal (pthread_t __thread1, pthread_t __thread2)
  __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int pthread_attr_init (pthread_attr_t *__attr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_destroy (pthread_attr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getdetachstate (const pthread_attr_t *__attr,
     int *__detachstate)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setdetachstate (pthread_attr_t *__attr,
     int __detachstate)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getguardsize (const pthread_attr_t *__attr,
          size_t *__guardsize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setguardsize (pthread_attr_t *__attr,
          size_t __guardsize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getschedparam (const pthread_attr_t *__restrict __attr,
           struct sched_param *__restrict __param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setschedparam (pthread_attr_t *__restrict __attr,
           const struct sched_param *__restrict
           __param) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_getschedpolicy (const pthread_attr_t *__restrict
     __attr, int *__restrict __policy)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setschedpolicy (pthread_attr_t *__attr, int __policy)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getinheritsched (const pthread_attr_t *__restrict
      __attr, int *__restrict __inherit)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setinheritsched (pthread_attr_t *__attr,
      int __inherit)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getscope (const pthread_attr_t *__restrict __attr,
      int *__restrict __scope)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setscope (pthread_attr_t *__attr, int __scope)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getstackaddr (const pthread_attr_t *__restrict
          __attr, void **__restrict __stackaddr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2))) __attribute__ ((__deprecated__));
extern int pthread_attr_setstackaddr (pthread_attr_t *__attr,
          void *__stackaddr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) __attribute__ ((__deprecated__));
extern int pthread_attr_getstacksize (const pthread_attr_t *__restrict
          __attr, size_t *__restrict __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setstacksize (pthread_attr_t *__attr,
          size_t __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getstack (const pthread_attr_t *__restrict __attr,
      void **__restrict __stackaddr,
      size_t *__restrict __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern int pthread_attr_setstack (pthread_attr_t *__attr, void *__stackaddr,
      size_t __stacksize) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 430 "/usr/include/pthread.h" 3 4
extern int pthread_setschedparam (pthread_t __target_thread, int __policy,
      const struct sched_param *__param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3)));
extern int pthread_getschedparam (pthread_t __target_thread,
      int *__restrict __policy,
      struct sched_param *__restrict __param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern int pthread_setschedprio (pthread_t __target_thread, int __prio)
     __attribute__ ((__nothrow__ , __leaf__));
# 495 "/usr/include/pthread.h" 3 4
extern int pthread_once (pthread_once_t *__once_control,
    void (*__init_routine) (void)) __attribute__ ((__nonnull__ (1, 2)));
# 507 "/usr/include/pthread.h" 3 4
extern int pthread_setcancelstate (int __state, int *__oldstate);
extern int pthread_setcanceltype (int __type, int *__oldtype);
extern int pthread_cancel (pthread_t __th);
extern void pthread_testcancel (void);
typedef struct
{
  struct
  {
    __jmp_buf __cancel_jmp_buf;
    int __mask_was_saved;
  } __cancel_jmp_buf[1];
  void *__pad[4];
} __pthread_unwind_buf_t __attribute__ ((__aligned__));
# 541 "/usr/include/pthread.h" 3 4
struct __pthread_cleanup_frame
{
  void (*__cancel_routine) (void *);
  void *__cancel_arg;
  int __do_it;
  int __cancel_type;
};
# 681 "/usr/include/pthread.h" 3 4
extern void __pthread_register_cancel (__pthread_unwind_buf_t *__buf)
     ;
# 693 "/usr/include/pthread.h" 3 4
extern void __pthread_unregister_cancel (__pthread_unwind_buf_t *__buf)
  ;
# 734 "/usr/include/pthread.h" 3 4
extern void __pthread_unwind_next (__pthread_unwind_buf_t *__buf)
     __attribute__ ((__noreturn__))
     __attribute__ ((__weak__))
     ;
struct __jmp_buf_tag;
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask) __attribute__ ((__nothrow__));
extern int pthread_mutex_init (pthread_mutex_t *__mutex,
          const pthread_mutexattr_t *__mutexattr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_destroy (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_trylock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_lock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_timedlock (pthread_mutex_t *__restrict __mutex,
        const struct timespec *__restrict
        __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutex_unlock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_getprioceiling (const pthread_mutex_t *
      __restrict __mutex,
      int *__restrict __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutex_setprioceiling (pthread_mutex_t *__restrict __mutex,
      int __prioceiling,
      int *__restrict __old_ceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 3)));
extern int pthread_mutex_consistent (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 807 "/usr/include/pthread.h" 3 4
extern int pthread_mutexattr_init (pthread_mutexattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_destroy (pthread_mutexattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getpshared (const pthread_mutexattr_t *
      __restrict __attr,
      int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setpshared (pthread_mutexattr_t *__attr,
      int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_gettype (const pthread_mutexattr_t *__restrict
          __attr, int *__restrict __kind)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_settype (pthread_mutexattr_t *__attr, int __kind)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getprotocol (const pthread_mutexattr_t *
       __restrict __attr,
       int *__restrict __protocol)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setprotocol (pthread_mutexattr_t *__attr,
       int __protocol)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getprioceiling (const pthread_mutexattr_t *
          __restrict __attr,
          int *__restrict __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setprioceiling (pthread_mutexattr_t *__attr,
          int __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getrobust (const pthread_mutexattr_t *__attr,
     int *__robustness)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setrobust (pthread_mutexattr_t *__attr,
     int __robustness)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 889 "/usr/include/pthread.h" 3 4
extern int pthread_rwlock_init (pthread_rwlock_t *__restrict __rwlock,
    const pthread_rwlockattr_t *__restrict
    __attr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_destroy (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_rdlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_tryrdlock (pthread_rwlock_t *__rwlock)
  __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_timedrdlock (pthread_rwlock_t *__restrict __rwlock,
           const struct timespec *__restrict
           __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlock_wrlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_trywrlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_timedwrlock (pthread_rwlock_t *__restrict __rwlock,
           const struct timespec *__restrict
           __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlock_unlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_init (pthread_rwlockattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_destroy (pthread_rwlockattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_getpshared (const pthread_rwlockattr_t *
       __restrict __attr,
       int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlockattr_setpshared (pthread_rwlockattr_t *__attr,
       int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_getkind_np (const pthread_rwlockattr_t *
       __restrict __attr,
       int *__restrict __pref)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlockattr_setkind_np (pthread_rwlockattr_t *__attr,
       int __pref) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_init (pthread_cond_t *__restrict __cond,
         const pthread_condattr_t *__restrict __cond_attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_destroy (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_signal (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_broadcast (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_wait (pthread_cond_t *__restrict __cond,
         pthread_mutex_t *__restrict __mutex)
     __attribute__ ((__nonnull__ (1, 2)));
# 1001 "/usr/include/pthread.h" 3 4
extern int pthread_cond_timedwait (pthread_cond_t *__restrict __cond,
       pthread_mutex_t *__restrict __mutex,
       const struct timespec *__restrict __abstime)
     __attribute__ ((__nonnull__ (1, 2, 3)));
extern int pthread_condattr_init (pthread_condattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_destroy (pthread_condattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_getpshared (const pthread_condattr_t *
     __restrict __attr,
     int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_condattr_setpshared (pthread_condattr_t *__attr,
     int __pshared) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_getclock (const pthread_condattr_t *
          __restrict __attr,
          __clockid_t *__restrict __clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_condattr_setclock (pthread_condattr_t *__attr,
          __clockid_t __clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1045 "/usr/include/pthread.h" 3 4
extern int pthread_spin_init (pthread_spinlock_t *__lock, int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_destroy (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_lock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_trylock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_unlock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_init (pthread_barrier_t *__restrict __barrier,
     const pthread_barrierattr_t *__restrict
     __attr, unsigned int __count)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_destroy (pthread_barrier_t *__barrier)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_wait (pthread_barrier_t *__barrier)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_init (pthread_barrierattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_destroy (pthread_barrierattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_getpshared (const pthread_barrierattr_t *
        __restrict __attr,
        int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_barrierattr_setpshared (pthread_barrierattr_t *__attr,
        int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1112 "/usr/include/pthread.h" 3 4
extern int pthread_key_create (pthread_key_t *__key,
          void (*__destr_function) (void *))
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_key_delete (pthread_key_t __key) __attribute__ ((__nothrow__ , __leaf__));
extern void *pthread_getspecific (pthread_key_t __key) __attribute__ ((__nothrow__ , __leaf__));
extern int pthread_setspecific (pthread_key_t __key,
    const void *__pointer) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int pthread_getcpuclockid (pthread_t __thread_id,
      __clockid_t *__clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 1146 "/usr/include/pthread.h" 3 4
extern int pthread_atfork (void (*__prepare) (void),
      void (*__parent) (void),
      void (*__child) (void)) __attribute__ ((__nothrow__ , __leaf__));
# 1160 "/usr/include/pthread.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_PTHREAD_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: setjmp.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/setjmp.h" 1 3 4
# 25 "/usr/include/setjmp.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/setjmp.h" 2 3 4
# 1 "/usr/include/bits/setjmp.h" 1 3 4
# 26 "/usr/include/bits/setjmp.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 27 "/usr/include/bits/setjmp.h" 2 3 4
# 31 "/usr/include/bits/setjmp.h" 3 4
typedef long int __jmp_buf[8];
# 30 "/usr/include/setjmp.h" 2 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 31 "/usr/include/setjmp.h" 2 3 4
struct __jmp_buf_tag
  {
    __jmp_buf __jmpbuf;
    int __mask_was_saved;
    __sigset_t __saved_mask;
  };
typedef struct __jmp_buf_tag jmp_buf[1];
extern int setjmp (jmp_buf __env) __attribute__ ((__nothrow__));
extern int __sigsetjmp (struct __jmp_buf_tag __env[1], int __savemask) __attribute__ ((__nothrow__));
extern int _setjmp (struct __jmp_buf_tag __env[1]) __attribute__ ((__nothrow__));
# 67 "/usr/include/setjmp.h" 3 4
extern void longjmp (struct __jmp_buf_tag __env[1], int __val)
     __attribute__ ((__nothrow__)) __attribute__ ((__noreturn__));
extern void _longjmp (struct __jmp_buf_tag __env[1], int __val)
     __attribute__ ((__nothrow__)) __attribute__ ((__noreturn__));
typedef struct __jmp_buf_tag sigjmp_buf[1];
# 93 "/usr/include/setjmp.h" 3 4
extern void siglongjmp (sigjmp_buf __env, int __val)
     __attribute__ ((__nothrow__)) __attribute__ ((__noreturn__));
# 103 "/usr/include/setjmp.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SETJMP_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/utsname.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/utsname.h" 1 3 4
# 25 "/usr/include/sys/utsname.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/utsname.h" 2 3 4
# 1 "/usr/include/bits/utsname.h" 1 3 4
# 30 "/usr/include/sys/utsname.h" 2 3 4
# 48 "/usr/include/sys/utsname.h" 3 4
# 48 "/usr/include/sys/utsname.h" 3 4
struct utsname
  {
    char sysname[65];
    char nodename[65];
    char release[65];
    char version[65];
    char machine[65];
    char __domainname[65];
  };
# 81 "/usr/include/sys/utsname.h" 3 4
extern int uname (struct utsname *__name) __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_UTSNAME_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: machine/endian.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: machine/endian.h: No such file or directory
 #include <machine/endian.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <machine/endian.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: machine/endian.h: No such file or directory
 #include <machine/endian.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: machine/endian.h: No such file or directory #include <machine/endian.h>          ^~~~~~~~~~~~~~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: limits.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 1 3 4
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/syslimits.h" 1 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 1 3 4
# 194 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 3 4
# 1 "/usr/include/limits.h" 1 3 4
# 26 "/usr/include/limits.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 27 "/usr/include/limits.h" 2 3 4
# 183 "/usr/include/limits.h" 3 4
# 1 "/usr/include/bits/posix1_lim.h" 1 3 4
# 27 "/usr/include/bits/posix1_lim.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/posix1_lim.h" 2 3 4
# 161 "/usr/include/bits/posix1_lim.h" 3 4
# 1 "/usr/include/bits/local_lim.h" 1 3 4
# 38 "/usr/include/bits/local_lim.h" 3 4
# 1 "/usr/include/linux/limits.h" 1 3 4
# 39 "/usr/include/bits/local_lim.h" 2 3 4
# 162 "/usr/include/bits/posix1_lim.h" 2 3 4
# 184 "/usr/include/limits.h" 2 3 4
# 1 "/usr/include/bits/posix2_lim.h" 1 3 4
# 188 "/usr/include/limits.h" 2 3 4
# 195 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 2 3 4
# 8 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/syslimits.h" 2 3 4
# 35 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_LIMITS_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: fcntl.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/fcntl.h" 1 3 4
# 25 "/usr/include/fcntl.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/fcntl.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 32 "/usr/include/fcntl.h" 2 3 4
# 1 "/usr/include/bits/fcntl.h" 1 3 4
# 35 "/usr/include/bits/fcntl.h" 3 4
struct flock
  {
    short int l_type;
    short int l_whence;
    __off_t l_start;
    __off_t l_len;
    __pid_t l_pid;
  };
# 61 "/usr/include/bits/fcntl.h" 3 4
# 1 "/usr/include/bits/fcntl-linux.h" 1 3 4
# 362 "/usr/include/bits/fcntl-linux.h" 3 4
# 436 "/usr/include/bits/fcntl-linux.h" 3 4
# 61 "/usr/include/bits/fcntl.h" 2 3 4
# 36 "/usr/include/fcntl.h" 2 3 4
# 50 "/usr/include/fcntl.h" 3 4
typedef __mode_t mode_t;
typedef __off_t off_t;
# 69 "/usr/include/fcntl.h" 3 4
typedef __pid_t pid_t;
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 76 "/usr/include/fcntl.h" 2 3 4
# 1 "/usr/include/bits/stat.h" 1 3 4
# 46 "/usr/include/bits/stat.h" 3 4
struct stat
  {
    __dev_t st_dev;
    __ino_t st_ino;
    __nlink_t st_nlink;
    __mode_t st_mode;
    __uid_t st_uid;
    __gid_t st_gid;
    int __pad0;
    __dev_t st_rdev;
    __off_t st_size;
    __blksize_t st_blksize;
    __blkcnt_t st_blocks;
# 91 "/usr/include/bits/stat.h" 3 4
    struct timespec st_atim;
    struct timespec st_mtim;
    struct timespec st_ctim;
# 106 "/usr/include/bits/stat.h" 3 4
    __syscall_slong_t __glibc_reserved[3];
# 115 "/usr/include/bits/stat.h" 3 4
  };
# 79 "/usr/include/fcntl.h" 2 3 4
# 175 "/usr/include/fcntl.h" 3 4
extern int fcntl (int __fd, int __cmd, ...);
# 195 "/usr/include/fcntl.h" 3 4
extern int open (const char *__file, int __oflag, ...) __attribute__ ((__nonnull__ (1)));
# 219 "/usr/include/fcntl.h" 3 4
extern int openat (int __fd, const char *__file, int __oflag, ...)
     __attribute__ ((__nonnull__ (2)));
# 241 "/usr/include/fcntl.h" 3 4
extern int creat (const char *__file, mode_t __mode) __attribute__ ((__nonnull__ (1)));
# 270 "/usr/include/fcntl.h" 3 4
extern int lockf (int __fd, int __cmd, off_t __len);
# 287 "/usr/include/fcntl.h" 3 4
extern int posix_fadvise (int __fd, off_t __offset, off_t __len,
     int __advise) __attribute__ ((__nothrow__ , __leaf__));
# 309 "/usr/include/fcntl.h" 3 4
extern int posix_fallocate (int __fd, off_t __offset, off_t __len);
# 331 "/usr/include/fcntl.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_FCNTL_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: string.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/string.h" 1 3 4
# 26 "/usr/include/string.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 27 "/usr/include/string.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 34 "/usr/include/string.h" 2 3 4
# 42 "/usr/include/string.h" 3 4
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
       size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memmove (void *__dest, const void *__src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memccpy (void *__restrict __dest, const void *__restrict __src,
        int __c, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void *memset (void *__s, int __c, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int memcmp (const void *__s1, const void *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 90 "/usr/include/string.h" 3 4
extern void *memchr (const void *__s, int __c, size_t __n)
      __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 121 "/usr/include/string.h" 3 4
extern char *strcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strncpy (char *__restrict __dest,
        const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strcat (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strncat (char *__restrict __dest, const char *__restrict __src,
        size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcmp (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strncmp (const char *__s1, const char *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcoll (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern size_t strxfrm (char *__restrict __dest,
         const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 153 "/usr/include/string.h" 2 3 4
extern int strcoll_l (const char *__s1, const char *__s2, locale_t __l)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern size_t strxfrm_l (char *__dest, const char *__src, size_t __n,
    locale_t __l) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4)));
extern char *strdup (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__nonnull__ (1)));
extern char *strndup (const char *__string, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__)) __attribute__ ((__nonnull__ (1)));
# 225 "/usr/include/string.h" 3 4
extern char *strchr (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 252 "/usr/include/string.h" 3 4
extern char *strrchr (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 272 "/usr/include/string.h" 3 4
extern size_t strcspn (const char *__s, const char *__reject)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern size_t strspn (const char *__s, const char *__accept)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 302 "/usr/include/string.h" 3 4
extern char *strpbrk (const char *__s, const char *__accept)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
# 329 "/usr/include/string.h" 3 4
extern char *strstr (const char *__haystack, const char *__needle)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strtok (char *__restrict __s, const char *__restrict __delim)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern char *__strtok_r (char *__restrict __s,
    const char *__restrict __delim,
    char **__restrict __save_ptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern char *strtok_r (char *__restrict __s, const char *__restrict __delim,
         char **__restrict __save_ptr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
# 384 "/usr/include/string.h" 3 4
extern size_t strlen (const char *__s)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern size_t strnlen (const char *__string, size_t __maxlen)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern char *strerror (int __errnum) __attribute__ ((__nothrow__ , __leaf__));
# 409 "/usr/include/string.h" 3 4
extern int strerror_r (int __errnum, char *__buf, size_t __buflen) __asm__ ("" "__xpg_strerror_r") __attribute__ ((__nothrow__ , __leaf__))
                        __attribute__ ((__nonnull__ (2)));
# 427 "/usr/include/string.h" 3 4
extern char *strerror_l (int __errnum, locale_t __l) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/strings.h" 1 3 4
# 23 "/usr/include/strings.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 24 "/usr/include/strings.h" 2 3 4
extern int bcmp (const void *__s1, const void *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bcopy (const void *__src, void *__dest, size_t __n)
  __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern void bzero (void *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 68 "/usr/include/strings.h" 3 4
extern char *index (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
# 96 "/usr/include/strings.h" 3 4
extern char *rindex (const char *__s, int __c)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1)));
extern int ffs (int __i) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int ffsl (long int __l) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
__extension__ extern int ffsll (long long int __ll)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int strcasecmp (const char *__s1, const char *__s2)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strncasecmp (const char *__s1, const char *__s2, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2)));
extern int strcasecmp_l (const char *__s1, const char *__s2, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern int strncasecmp_l (const char *__s1, const char *__s2,
     size_t __n, locale_t __loc)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__pure__)) __attribute__ ((__nonnull__ (1, 2, 4)));
# 432 "/usr/include/string.h" 2 3 4
extern void explicit_bzero (void *__s, size_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern char *strsep (char **__restrict __stringp,
       const char *__restrict __delim)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *strsignal (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern char *__stpcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *stpcpy (char *__restrict __dest, const char *__restrict __src)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *__stpncpy (char *__restrict __dest,
   const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern char *stpncpy (char *__restrict __dest,
        const char *__restrict __src, size_t __n)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
# 498 "/usr/include/string.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_STRING_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/times.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/times.h" 1 3 4
# 25 "/usr/include/sys/times.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/times.h" 2 3 4
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 5 "/usr/include/bits/types/clock_t.h" 2 3 4
typedef __clock_t clock_t;
# 28 "/usr/include/sys/times.h" 2 3 4
struct tms
  {
    clock_t tms_utime;
    clock_t tms_stime;
    clock_t tms_cutime;
    clock_t tms_cstime;
  };
extern clock_t times (struct tms *__buffer) __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_TIMES_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: io.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: io.h: No such file or directory
 #include <io.h>
          ^~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <io.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: io.h: No such file or directory
 #include <io.h>
          ^~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: io.h: No such file or directory #include <io.h>          ^~~~~~compilation terminated.:
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: stdint.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 1 3 4
# 9 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 26 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 33 "/usr/include/bits/libc-header-start.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 34 "/usr/include/bits/libc-header-start.h" 2 3 4
# 27 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wchar.h" 1 3 4
# 29 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 30 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 35 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 24 "/usr/include/bits/stdint-uintn.h" 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 38 "/usr/include/stdint.h" 2 3 4
typedef __int_least8_t int_least8_t;
typedef __int_least16_t int_least16_t;
typedef __int_least32_t int_least32_t;
typedef __int_least64_t int_least64_t;
typedef __uint_least8_t uint_least8_t;
typedef __uint_least16_t uint_least16_t;
typedef __uint_least32_t uint_least32_t;
typedef __uint_least64_t uint_least64_t;
typedef signed char int_fast8_t;
typedef long int int_fast16_t;
typedef long int int_fast32_t;
typedef long int int_fast64_t;
# 71 "/usr/include/stdint.h" 3 4
typedef unsigned char uint_fast8_t;
typedef unsigned long int uint_fast16_t;
typedef unsigned long int uint_fast32_t;
typedef unsigned long int uint_fast64_t;
# 87 "/usr/include/stdint.h" 3 4
typedef long int intptr_t;
typedef unsigned long int uintptr_t;
# 101 "/usr/include/stdint.h" 3 4
typedef __intmax_t intmax_t;
typedef __uintmax_t uintmax_t;
# 10 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_STDINT_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: pwd.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/pwd.h" 1 3 4
# 25 "/usr/include/pwd.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/pwd.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/pwd.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 33 "/usr/include/pwd.h" 2 3 4
typedef __gid_t gid_t;
typedef __uid_t uid_t;
struct passwd
{
  char *pw_name;
  char *pw_passwd;
  __uid_t pw_uid;
  __gid_t pw_gid;
  char *pw_gecos;
  char *pw_dir;
  char *pw_shell;
};
# 1 "/usr/include/bits/types/FILE.h" 1 3 4
struct _IO_FILE;
typedef struct _IO_FILE FILE;
# 64 "/usr/include/pwd.h" 2 3 4
# 72 "/usr/include/pwd.h" 3 4
extern void setpwent (void);
extern void endpwent (void);
extern struct passwd *getpwent (void);
# 94 "/usr/include/pwd.h" 3 4
extern struct passwd *fgetpwent (FILE *__stream) __attribute__ ((__nonnull__ (1)));
extern int putpwent (const struct passwd *__restrict __p,
       FILE *__restrict __f);
extern struct passwd *getpwuid (__uid_t __uid);
extern struct passwd *getpwnam (const char *__name) __attribute__ ((__nonnull__ (1)));
# 139 "/usr/include/pwd.h" 3 4
extern int getpwent_r (struct passwd *__restrict __resultbuf,
         char *__restrict __buffer, size_t __buflen,
         struct passwd **__restrict __result)
         __attribute__ ((__nonnull__ (1, 2, 4)));
extern int getpwuid_r (__uid_t __uid,
         struct passwd *__restrict __resultbuf,
         char *__restrict __buffer, size_t __buflen,
         struct passwd **__restrict __result)
         __attribute__ ((__nonnull__ (2, 3, 5)));
extern int getpwnam_r (const char *__restrict __name,
         struct passwd *__restrict __resultbuf,
         char *__restrict __buffer, size_t __buflen,
         struct passwd **__restrict __result)
         __attribute__ ((__nonnull__ (1, 2, 3, 5)));
# 166 "/usr/include/pwd.h" 3 4
extern int fgetpwent_r (FILE *__restrict __stream,
   struct passwd *__restrict __resultbuf,
   char *__restrict __buffer, size_t __buflen,
   struct passwd **__restrict __result)
   __attribute__ ((__nonnull__ (1, 2, 3, 5)));
# 187 "/usr/include/pwd.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_PWD_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: float.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/float.h" 1 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_FLOAT_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/param.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/param.h" 1 3 4
# 23 "/usr/include/sys/param.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 24 "/usr/include/sys/param.h" 2 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 25 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 26 "/usr/include/sys/param.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 1 3 4
# 34 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/syslimits.h" 1 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 1 3 4
# 194 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 3 4
# 1 "/usr/include/limits.h" 1 3 4
# 26 "/usr/include/limits.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 27 "/usr/include/limits.h" 2 3 4
# 183 "/usr/include/limits.h" 3 4
# 1 "/usr/include/bits/posix1_lim.h" 1 3 4
# 27 "/usr/include/bits/posix1_lim.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/posix1_lim.h" 2 3 4
# 161 "/usr/include/bits/posix1_lim.h" 3 4
# 1 "/usr/include/bits/local_lim.h" 1 3 4
# 38 "/usr/include/bits/local_lim.h" 3 4
# 1 "/usr/include/linux/limits.h" 1 3 4
# 39 "/usr/include/bits/local_lim.h" 2 3 4
# 162 "/usr/include/bits/posix1_lim.h" 2 3 4
# 184 "/usr/include/limits.h" 2 3 4
# 1 "/usr/include/bits/posix2_lim.h" 1 3 4
# 188 "/usr/include/limits.h" 2 3 4
# 195 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 2 3 4
# 8 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/syslimits.h" 2 3 4
# 35 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed/limits.h" 2 3 4
# 27 "/usr/include/sys/param.h" 2 3 4
# 1 "/usr/include/signal.h" 1 3 4
# 27 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/signum.h" 1 3 4
# 26 "/usr/include/bits/signum.h" 3 4
# 1 "/usr/include/bits/signum-generic.h" 1 3 4
# 27 "/usr/include/bits/signum.h" 2 3 4
# 31 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sig_atomic_t.h" 1 3 4
typedef __sig_atomic_t sig_atomic_t;
# 33 "/usr/include/signal.h" 2 3 4
# 57 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/types/siginfo_t.h" 1 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 5 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 1 "/usr/include/bits/types/__sigval_t.h" 1 3 4
# 24 "/usr/include/bits/types/__sigval_t.h" 3 4
union sigval
{
  int sival_int;
  void *sival_ptr;
};
typedef union sigval __sigval_t;
# 7 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 16 "/usr/include/bits/types/siginfo_t.h" 3 4
# 1 "/usr/include/bits/siginfo-arch.h" 1 3 4
# 17 "/usr/include/bits/types/siginfo_t.h" 2 3 4
# 36 "/usr/include/bits/types/siginfo_t.h" 3 4
typedef struct
  {
    int si_signo;
    int si_errno;
    int si_code;
    int __pad0;
    union
      {
 int _pad[((128 / sizeof (int)) - 4)];
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
   } _kill;
 struct
   {
     int si_tid;
     int si_overrun;
     __sigval_t si_sigval;
   } _timer;
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
     __sigval_t si_sigval;
   } _rt;
 struct
   {
     __pid_t si_pid;
     __uid_t si_uid;
     int si_status;
     __clock_t si_utime;
     __clock_t si_stime;
   } _sigchld;
 struct
   {
     void *si_addr;
    
     short int si_addr_lsb;
     union
       {
  struct
    {
      void *_lower;
      void *_upper;
    } _addr_bnd;
  __uint32_t _pkey;
       } _bounds;
   } _sigfault;
 struct
   {
     long int si_band;
     int si_fd;
   } _sigpoll;
 struct
   {
     void *_call_addr;
     int _syscall;
     unsigned int _arch;
   } _sigsys;
      } _sifields;
  } siginfo_t ;
# 58 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/siginfo-consts.h" 1 3 4
# 35 "/usr/include/bits/siginfo-consts.h" 3 4
enum
{
  SI_ASYNCNL = -60,
  SI_TKILL = -6,
  SI_SIGIO,
  SI_ASYNCIO,
  SI_MESGQ,
  SI_TIMER,
  SI_QUEUE,
  SI_USER,
  SI_KERNEL = 0x80
# 63 "/usr/include/bits/siginfo-consts.h" 3 4
};
enum
{
  ILL_ILLOPC = 1,
  ILL_ILLOPN,
  ILL_ILLADR,
  ILL_ILLTRP,
  ILL_PRVOPC,
  ILL_PRVREG,
  ILL_COPROC,
  ILL_BADSTK
};
enum
{
  FPE_INTDIV = 1,
  FPE_INTOVF,
  FPE_FLTDIV,
  FPE_FLTOVF,
  FPE_FLTUND,
  FPE_FLTRES,
  FPE_FLTINV,
  FPE_FLTSUB
};
enum
{
  SEGV_MAPERR = 1,
  SEGV_ACCERR,
  SEGV_BNDERR,
  SEGV_PKUERR
};
enum
{
  BUS_ADRALN = 1,
  BUS_ADRERR,
  BUS_OBJERR,
  BUS_MCEERR_AR,
  BUS_MCEERR_AO
};
# 151 "/usr/include/bits/siginfo-consts.h" 3 4
enum
{
  CLD_EXITED = 1,
  CLD_KILLED,
  CLD_DUMPED,
  CLD_TRAPPED,
  CLD_STOPPED,
  CLD_CONTINUED
};
enum
{
  POLL_IN = 1,
  POLL_OUT,
  POLL_MSG,
  POLL_ERR,
  POLL_PRI,
  POLL_HUP
};
# 59 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sigval_t.h" 1 3 4
# 16 "/usr/include/bits/types/sigval_t.h" 3 4
typedef __sigval_t sigval_t;
# 63 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/sigevent_t.h" 1 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 5 "/usr/include/bits/types/sigevent_t.h" 2 3 4
# 22 "/usr/include/bits/types/sigevent_t.h" 3 4
typedef struct sigevent
  {
    __sigval_t sigev_value;
    int sigev_signo;
    int sigev_notify;
    union
      {
 int _pad[((64 / sizeof (int)) - 4)];
 __pid_t _tid;
 struct
   {
     void (*_function) (__sigval_t);
     pthread_attr_t *_attribute;
   } _sigev_thread;
      } _sigev_un;
  } sigevent_t;
# 67 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/sigevent-consts.h" 1 3 4
# 27 "/usr/include/bits/sigevent-consts.h" 3 4
enum
{
  SIGEV_SIGNAL = 0,
  SIGEV_NONE,
  SIGEV_THREAD,
  SIGEV_THREAD_ID = 4
};
# 68 "/usr/include/signal.h" 2 3 4
typedef void (*__sighandler_t) (int);
extern __sighandler_t __sysv_signal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
# 88 "/usr/include/signal.h" 3 4
extern __sighandler_t signal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
# 112 "/usr/include/signal.h" 3 4
extern int kill (__pid_t __pid, int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern int killpg (__pid_t __pgrp, int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern int raise (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern __sighandler_t ssignal (int __sig, __sighandler_t __handler)
     __attribute__ ((__nothrow__ , __leaf__));
extern int gsignal (int __sig) __attribute__ ((__nothrow__ , __leaf__));
extern void psignal (int __sig, const char *__s);
extern void psiginfo (const siginfo_t *__pinfo, const char *__s);
# 170 "/usr/include/signal.h" 3 4
extern int sigblock (int __mask) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
extern int sigsetmask (int __mask) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
extern int siggetmask (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
# 190 "/usr/include/signal.h" 3 4
typedef __sighandler_t sig_t;
extern int sigemptyset (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigfillset (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigaddset (sigset_t *__set, int __signo) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigdelset (sigset_t *__set, int __signo) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigismember (const sigset_t *__set, int __signo)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 226 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/sigaction.h" 1 3 4
# 27 "/usr/include/bits/sigaction.h" 3 4
struct sigaction
  {
    union
      {
 __sighandler_t sa_handler;
 void (*sa_sigaction) (int, siginfo_t *, void *);
      }
    __sigaction_handler;
    __sigset_t sa_mask;
    int sa_flags;
    void (*sa_restorer) (void);
  };
# 227 "/usr/include/signal.h" 2 3 4
extern int sigprocmask (int __how, const sigset_t *__restrict __set,
   sigset_t *__restrict __oset) __attribute__ ((__nothrow__ , __leaf__));
extern int sigsuspend (const sigset_t *__set) __attribute__ ((__nonnull__ (1)));
extern int sigaction (int __sig, const struct sigaction *__restrict __act,
        struct sigaction *__restrict __oact) __attribute__ ((__nothrow__ , __leaf__));
extern int sigpending (sigset_t *__set) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sigwait (const sigset_t *__restrict __set, int *__restrict __sig)
     __attribute__ ((__nonnull__ (1, 2)));
extern int sigwaitinfo (const sigset_t *__restrict __set,
   siginfo_t *__restrict __info) __attribute__ ((__nonnull__ (1)));
extern int sigtimedwait (const sigset_t *__restrict __set,
    siginfo_t *__restrict __info,
    const struct timespec *__restrict __timeout)
     __attribute__ ((__nonnull__ (1)));
extern int sigqueue (__pid_t __pid, int __sig, const union sigval __val)
     __attribute__ ((__nothrow__ , __leaf__));
# 286 "/usr/include/signal.h" 3 4
extern const char *const _sys_siglist[(64 + 1)];
extern const char *const sys_siglist[(64 + 1)];
# 1 "/usr/include/bits/sigcontext.h" 1 3 4
# 31 "/usr/include/bits/sigcontext.h" 3 4
struct _fpx_sw_bytes
{
  __uint32_t magic1;
  __uint32_t extended_size;
  __uint64_t xstate_bv;
  __uint32_t xstate_size;
  __uint32_t __glibc_reserved1[7];
};
struct _fpreg
{
  unsigned short significand[4];
  unsigned short exponent;
};
struct _fpxreg
{
  unsigned short significand[4];
  unsigned short exponent;
  unsigned short __glibc_reserved1[3];
};
struct _xmmreg
{
  __uint32_t element[4];
};
# 123 "/usr/include/bits/sigcontext.h" 3 4
struct _fpstate
{
  __uint16_t cwd;
  __uint16_t swd;
  __uint16_t ftw;
  __uint16_t fop;
  __uint64_t rip;
  __uint64_t rdp;
  __uint32_t mxcsr;
  __uint32_t mxcr_mask;
  struct _fpxreg _st[8];
  struct _xmmreg _xmm[16];
  __uint32_t __glibc_reserved1[24];
};
struct sigcontext
{
  __uint64_t r8;
  __uint64_t r9;
  __uint64_t r10;
  __uint64_t r11;
  __uint64_t r12;
  __uint64_t r13;
  __uint64_t r14;
  __uint64_t r15;
  __uint64_t rdi;
  __uint64_t rsi;
  __uint64_t rbp;
  __uint64_t rbx;
  __uint64_t rdx;
  __uint64_t rax;
  __uint64_t rcx;
  __uint64_t rsp;
  __uint64_t rip;
  __uint64_t eflags;
  unsigned short cs;
  unsigned short gs;
  unsigned short fs;
  unsigned short __pad0;
  __uint64_t err;
  __uint64_t trapno;
  __uint64_t oldmask;
  __uint64_t cr2;
  __extension__ union
    {
      struct _fpstate * fpstate;
      __uint64_t __fpstate_word;
    };
  __uint64_t __reserved1 [8];
};
struct _xsave_hdr
{
  __uint64_t xstate_bv;
  __uint64_t __glibc_reserved1[2];
  __uint64_t __glibc_reserved2[5];
};
struct _ymmh_state
{
  __uint32_t ymmh_space[64];
};
struct _xstate
{
  struct _fpstate fpstate;
  struct _xsave_hdr xstate_hdr;
  struct _ymmh_state ymmh;
};
# 292 "/usr/include/signal.h" 2 3 4
extern int sigreturn (struct sigcontext *__scp) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 302 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/types/stack_t.h" 1 3 4
# 23 "/usr/include/bits/types/stack_t.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 24 "/usr/include/bits/types/stack_t.h" 2 3 4
typedef struct
  {
    void *ss_sp;
    int ss_flags;
    size_t ss_size;
  } stack_t;
# 304 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/sys/ucontext.h" 1 3 4
# 37 "/usr/include/sys/ucontext.h" 3 4
__extension__ typedef long long int greg_t;
# 46 "/usr/include/sys/ucontext.h" 3 4
typedef greg_t gregset_t[23];
# 101 "/usr/include/sys/ucontext.h" 3 4
struct _libc_fpxreg
{
  unsigned short int significand[4];
  unsigned short int exponent;
  unsigned short int __glibc_reserved1[3];
};
struct _libc_xmmreg
{
  __uint32_t element[4];
};
struct _libc_fpstate
{
  __uint16_t cwd;
  __uint16_t swd;
  __uint16_t ftw;
  __uint16_t fop;
  __uint64_t rip;
  __uint64_t rdp;
  __uint32_t mxcsr;
  __uint32_t mxcr_mask;
  struct _libc_fpxreg _st[8];
  struct _libc_xmmreg _xmm[16];
  __uint32_t __glibc_reserved1[24];
};
typedef struct _libc_fpstate *fpregset_t;
typedef struct
  {
    gregset_t gregs;
    fpregset_t fpregs;
    __extension__ unsigned long long __reserved1 [8];
} mcontext_t;
typedef struct ucontext_t
  {
    unsigned long int uc_flags;
    struct ucontext_t *uc_link;
    stack_t uc_stack;
    mcontext_t uc_mcontext;
    sigset_t uc_sigmask;
    struct _libc_fpstate __fpregs_mem;
    __extension__ unsigned long long int __ssp[4];
  } ucontext_t;
# 307 "/usr/include/signal.h" 2 3 4
extern int siginterrupt (int __sig, int __interrupt) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/bits/sigstack.h" 1 3 4
# 317 "/usr/include/signal.h" 2 3 4
# 1 "/usr/include/bits/ss_flags.h" 1 3 4
# 27 "/usr/include/bits/ss_flags.h" 3 4
enum
{
  SS_ONSTACK = 1,
  SS_DISABLE
};
# 318 "/usr/include/signal.h" 2 3 4
extern int sigaltstack (const stack_t *__restrict __ss,
   stack_t *__restrict __oss) __attribute__ ((__nothrow__ , __leaf__));
# 1 "/usr/include/bits/types/struct_sigstack.h" 1 3 4
# 23 "/usr/include/bits/types/struct_sigstack.h" 3 4
struct sigstack
  {
    void *ss_sp;
    int ss_onstack;
  };
# 328 "/usr/include/signal.h" 2 3 4
extern int sigstack (struct sigstack *__ss, struct sigstack *__oss)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__deprecated__));
# 359 "/usr/include/signal.h" 3 4
# 1 "/usr/include/bits/sigthread.h" 1 3 4
# 31 "/usr/include/bits/sigthread.h" 3 4
extern int pthread_sigmask (int __how,
       const __sigset_t *__restrict __newmask,
       __sigset_t *__restrict __oldmask)__attribute__ ((__nothrow__ , __leaf__));
extern int pthread_kill (pthread_t __threadid, int __signo) __attribute__ ((__nothrow__ , __leaf__));
# 360 "/usr/include/signal.h" 2 3 4
extern int __libc_current_sigrtmin (void) __attribute__ ((__nothrow__ , __leaf__));
extern int __libc_current_sigrtmax (void) __attribute__ ((__nothrow__ , __leaf__));
# 29 "/usr/include/sys/param.h" 2 3 4
# 1 "/usr/include/bits/param.h" 1 3 4
# 28 "/usr/include/bits/param.h" 3 4
# 1 "/usr/include/linux/param.h" 1 3 4
# 1 "/usr/include/asm/param.h" 1 3 4
# 1 "/usr/include/asm-generic/param.h" 1 3 4
# 1 "/usr/include/asm/param.h" 2 3 4
# 6 "/usr/include/linux/param.h" 2 3 4
# 29 "/usr/include/bits/param.h" 2 3 4
# 32 "/usr/include/sys/param.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_PARAM_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: netdb.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/netdb.h" 1 3 4
# 25 "/usr/include/netdb.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/netdb.h" 2 3 4
# 1 "/usr/include/netinet/in.h" 1 3 4
# 22 "/usr/include/netinet/in.h" 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 22 "/usr/include/bits/stdint-uintn.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 23 "/usr/include/bits/stdint-uintn.h" 2 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 23 "/usr/include/netinet/in.h" 2 3 4
# 1 "/usr/include/sys/socket.h" 1 3 4
# 24 "/usr/include/sys/socket.h" 3 4
# 1 "/usr/include/bits/types/struct_iovec.h" 1 3 4
# 23 "/usr/include/bits/types/struct_iovec.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 24 "/usr/include/bits/types/struct_iovec.h" 2 3 4
struct iovec
  {
    void *iov_base;
    size_t iov_len;
  };
# 27 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 29 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/socket.h" 1 3 4
# 27 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 28 "/usr/include/bits/socket.h" 2 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 30 "/usr/include/bits/socket.h" 2 3 4
typedef __socklen_t socklen_t;
# 1 "/usr/include/bits/socket_type.h" 1 3 4
# 24 "/usr/include/bits/socket_type.h" 3 4
enum __socket_type
{
  SOCK_STREAM = 1,
  SOCK_DGRAM = 2,
  SOCK_RAW = 3,
  SOCK_RDM = 4,
  SOCK_SEQPACKET = 5,
  SOCK_DCCP = 6,
  SOCK_PACKET = 10,
  SOCK_CLOEXEC = 02000000,
  SOCK_NONBLOCK = 00004000
};
# 39 "/usr/include/bits/socket.h" 2 3 4
# 172 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/bits/sockaddr.h" 1 3 4
# 28 "/usr/include/bits/sockaddr.h" 3 4
typedef unsigned short int sa_family_t;
# 173 "/usr/include/bits/socket.h" 2 3 4
struct sockaddr
  {
    sa_family_t sa_family;
    char sa_data[14];
  };
# 188 "/usr/include/bits/socket.h" 3 4
struct sockaddr_storage
  {
    sa_family_t ss_family;
    char __ss_padding[(128 - (sizeof (unsigned short int)) - sizeof (unsigned long int))];
    unsigned long int __ss_align;
  };
enum
  {
    MSG_OOB = 0x01,
    MSG_PEEK = 0x02,
    MSG_DONTROUTE = 0x04,
    MSG_CTRUNC = 0x08,
    MSG_PROXY = 0x10,
    MSG_TRUNC = 0x20,
    MSG_DONTWAIT = 0x40,
    MSG_EOR = 0x80,
    MSG_WAITALL = 0x100,
    MSG_FIN = 0x200,
    MSG_SYN = 0x400,
    MSG_CONFIRM = 0x800,
    MSG_RST = 0x1000,
    MSG_ERRQUEUE = 0x2000,
    MSG_NOSIGNAL = 0x4000,
    MSG_MORE = 0x8000,
    MSG_WAITFORONE = 0x10000,
    MSG_BATCH = 0x40000,
    MSG_ZEROCOPY = 0x4000000,
    MSG_FASTOPEN = 0x20000000,
    MSG_CMSG_CLOEXEC = 0x40000000
  };
struct msghdr
  {
    void *msg_name;
    socklen_t msg_namelen;
    struct iovec *msg_iov;
    size_t msg_iovlen;
    void *msg_control;
    size_t msg_controllen;
    int msg_flags;
  };
struct cmsghdr
  {
    size_t cmsg_len;
    int cmsg_level;
    int cmsg_type;
    __extension__ unsigned char __cmsg_data [];
  };
# 302 "/usr/include/bits/socket.h" 3 4
extern struct cmsghdr *__cmsg_nxthdr (struct msghdr *__mhdr,
          struct cmsghdr *__cmsg) __attribute__ ((__nothrow__ , __leaf__));
# 329 "/usr/include/bits/socket.h" 3 4
enum
  {
    SCM_RIGHTS = 0x01
  };
# 390 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/asm/socket.h" 1 3 4
# 1 "/usr/include/asm-generic/socket.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 1 3 4
# 1 "/usr/include/asm-generic/sockios.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 2 3 4
# 6 "/usr/include/asm-generic/socket.h" 2 3 4
# 1 "/usr/include/asm/socket.h" 2 3 4
# 391 "/usr/include/bits/socket.h" 2 3 4
# 444 "/usr/include/bits/socket.h" 3 4
struct linger
  {
    int l_onoff;
    int l_linger;
  };
# 34 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/types/struct_osockaddr.h" 1 3 4
struct osockaddr
{
  unsigned short int sa_family;
  unsigned char sa_data[14];
};
# 37 "/usr/include/sys/socket.h" 2 3 4
enum
{
  SHUT_RD = 0,
  SHUT_WR,
  SHUT_RDWR
};
# 102 "/usr/include/sys/socket.h" 3 4
extern int socket (int __domain, int __type, int __protocol) __attribute__ ((__nothrow__ , __leaf__));
extern int socketpair (int __domain, int __type, int __protocol,
         int __fds[2]) __attribute__ ((__nothrow__ , __leaf__));
extern int bind (int __fd, const struct sockaddr * __addr, socklen_t __len)
     __attribute__ ((__nothrow__ , __leaf__));
extern int getsockname (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
# 126 "/usr/include/sys/socket.h" 3 4
extern int connect (int __fd, const struct sockaddr * __addr, socklen_t __len);
extern int getpeername (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
extern ssize_t send (int __fd, const void *__buf, size_t __n, int __flags);
extern ssize_t recv (int __fd, void *__buf, size_t __n, int __flags);
extern ssize_t sendto (int __fd, const void *__buf, size_t __n,
         int __flags, const struct sockaddr * __addr,
         socklen_t __addr_len);
# 163 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvfrom (int __fd, void *__restrict __buf, size_t __n,
    int __flags, struct sockaddr *__restrict __addr,
    socklen_t *__restrict __addr_len);
extern ssize_t sendmsg (int __fd, const struct msghdr *__message,
   int __flags);
# 191 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvmsg (int __fd, struct msghdr *__message, int __flags);
# 208 "/usr/include/sys/socket.h" 3 4
extern int getsockopt (int __fd, int __level, int __optname,
         void *__restrict __optval,
         socklen_t *__restrict __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int setsockopt (int __fd, int __level, int __optname,
         const void *__optval, socklen_t __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int listen (int __fd, int __n) __attribute__ ((__nothrow__ , __leaf__));
# 232 "/usr/include/sys/socket.h" 3 4
extern int accept (int __fd, struct sockaddr *__restrict __addr,
     socklen_t *__restrict __addr_len);
# 250 "/usr/include/sys/socket.h" 3 4
extern int shutdown (int __fd, int __how) __attribute__ ((__nothrow__ , __leaf__));
extern int sockatmark (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int isfdtype (int __fd, int __fdtype) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/sys/socket.h" 3 4
# 24 "/usr/include/netinet/in.h" 2 3 4
typedef uint32_t in_addr_t;
struct in_addr
  {
    in_addr_t s_addr;
  };
# 1 "/usr/include/bits/in.h" 1 3 4
# 142 "/usr/include/bits/in.h" 3 4
struct ip_opts
  {
    struct in_addr ip_dst;
    char ip_opts[40];
  };
struct ip_mreqn
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_address;
    int imr_ifindex;
  };
struct in_pktinfo
  {
    int ipi_ifindex;
    struct in_addr ipi_spec_dst;
    struct in_addr ipi_addr;
  };
# 38 "/usr/include/netinet/in.h" 2 3 4
enum
  {
    IPPROTO_IP = 0,
    IPPROTO_ICMP = 1,
    IPPROTO_IGMP = 2,
    IPPROTO_IPIP = 4,
    IPPROTO_TCP = 6,
    IPPROTO_EGP = 8,
    IPPROTO_PUP = 12,
    IPPROTO_UDP = 17,
    IPPROTO_IDP = 22,
    IPPROTO_TP = 29,
    IPPROTO_DCCP = 33,
    IPPROTO_IPV6 = 41,
    IPPROTO_RSVP = 46,
    IPPROTO_GRE = 47,
    IPPROTO_ESP = 50,
    IPPROTO_AH = 51,
    IPPROTO_MTP = 92,
    IPPROTO_BEETPH = 94,
    IPPROTO_ENCAP = 98,
    IPPROTO_PIM = 103,
    IPPROTO_COMP = 108,
    IPPROTO_SCTP = 132,
    IPPROTO_UDPLITE = 136,
    IPPROTO_MPLS = 137,
    IPPROTO_RAW = 255,
    IPPROTO_MAX
  };
enum
  {
    IPPROTO_HOPOPTS = 0,
    IPPROTO_ROUTING = 43,
    IPPROTO_FRAGMENT = 44,
    IPPROTO_ICMPV6 = 58,
    IPPROTO_NONE = 59,
    IPPROTO_DSTOPTS = 60,
    IPPROTO_MH = 135
  };
typedef uint16_t in_port_t;
enum
  {
    IPPORT_ECHO = 7,
    IPPORT_DISCARD = 9,
    IPPORT_SYSTAT = 11,
    IPPORT_DAYTIME = 13,
    IPPORT_NETSTAT = 15,
    IPPORT_FTP = 21,
    IPPORT_TELNET = 23,
    IPPORT_SMTP = 25,
    IPPORT_TIMESERVER = 37,
    IPPORT_NAMESERVER = 42,
    IPPORT_WHOIS = 43,
    IPPORT_MTP = 57,
    IPPORT_TFTP = 69,
    IPPORT_RJE = 77,
    IPPORT_FINGER = 79,
    IPPORT_TTYLINK = 87,
    IPPORT_SUPDUP = 95,
    IPPORT_EXECSERVER = 512,
    IPPORT_LOGINSERVER = 513,
    IPPORT_CMDSERVER = 514,
    IPPORT_EFSSERVER = 520,
    IPPORT_BIFFUDP = 512,
    IPPORT_WHOSERVER = 513,
    IPPORT_ROUTESERVER = 520,
    IPPORT_RESERVED = 1024,
    IPPORT_USERRESERVED = 5000
  };
# 211 "/usr/include/netinet/in.h" 3 4
struct in6_addr
  {
    union
      {
 uint8_t __u6_addr8[16];
 uint16_t __u6_addr16[8];
 uint32_t __u6_addr32[4];
      } __in6_u;
  };
extern const struct in6_addr in6addr_any;
extern const struct in6_addr in6addr_loopback;
# 237 "/usr/include/netinet/in.h" 3 4
struct sockaddr_in
  {
    sa_family_t sin_family;
    in_port_t sin_port;
    struct in_addr sin_addr;
    unsigned char sin_zero[sizeof (struct sockaddr) -
      (sizeof (unsigned short int)) -
      sizeof (in_port_t) -
      sizeof (struct in_addr)];
  };
struct sockaddr_in6
  {
    sa_family_t sin6_family;
    in_port_t sin6_port;
    uint32_t sin6_flowinfo;
    struct in6_addr sin6_addr;
    uint32_t sin6_scope_id;
  };
struct ip_mreq
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_interface;
  };
struct ip_mreq_source
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_interface;
    struct in_addr imr_sourceaddr;
  };
struct ipv6_mreq
  {
    struct in6_addr ipv6mr_multiaddr;
    unsigned int ipv6mr_interface;
  };
struct group_req
  {
    uint32_t gr_interface;
    struct sockaddr_storage gr_group;
  };
struct group_source_req
  {
    uint32_t gsr_interface;
    struct sockaddr_storage gsr_group;
    struct sockaddr_storage gsr_source;
  };
struct ip_msfilter
  {
    struct in_addr imsf_multiaddr;
    struct in_addr imsf_interface;
    uint32_t imsf_fmode;
    uint32_t imsf_numsrc;
    struct in_addr imsf_slist[1];
  };
struct group_filter
  {
    uint32_t gf_interface;
    struct sockaddr_storage gf_group;
    uint32_t gf_fmode;
    uint32_t gf_numsrc;
    struct sockaddr_storage gf_slist[1];
};
# 374 "/usr/include/netinet/in.h" 3 4
extern uint32_t ntohl (uint32_t __netlong) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint16_t ntohs (uint16_t __netshort)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint32_t htonl (uint32_t __hostlong)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint16_t htons (uint16_t __hostshort)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 386 "/usr/include/netinet/in.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 387 "/usr/include/netinet/in.h" 2 3 4
# 502 "/usr/include/netinet/in.h" 3 4
extern int bindresvport (int __sockfd, struct sockaddr_in *__sock_in) __attribute__ ((__nothrow__ , __leaf__));
extern int bindresvport6 (int __sockfd, struct sockaddr_in6 *__sock_in)
     __attribute__ ((__nothrow__ , __leaf__));
# 630 "/usr/include/netinet/in.h" 3 4
# 28 "/usr/include/netdb.h" 2 3 4
# 1 "/usr/include/rpc/netdb.h" 1 3 4
# 42 "/usr/include/rpc/netdb.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 43 "/usr/include/rpc/netdb.h" 2 3 4
struct rpcent
{
  char *r_name;
  char **r_aliases;
  int r_number;
};
extern void setrpcent (int __stayopen) __attribute__ ((__nothrow__ , __leaf__));
extern void endrpcent (void) __attribute__ ((__nothrow__ , __leaf__));
extern struct rpcent *getrpcbyname (const char *__name) __attribute__ ((__nothrow__ , __leaf__));
extern struct rpcent *getrpcbynumber (int __number) __attribute__ ((__nothrow__ , __leaf__));
extern struct rpcent *getrpcent (void) __attribute__ ((__nothrow__ , __leaf__));
extern int getrpcbyname_r (const char *__name, struct rpcent *__result_buf,
      char *__buffer, size_t __buflen,
      struct rpcent **__result) __attribute__ ((__nothrow__ , __leaf__));
extern int getrpcbynumber_r (int __number, struct rpcent *__result_buf,
        char *__buffer, size_t __buflen,
        struct rpcent **__result) __attribute__ ((__nothrow__ , __leaf__));
extern int getrpcent_r (struct rpcent *__result_buf, char *__buffer,
   size_t __buflen, struct rpcent **__result) __attribute__ ((__nothrow__ , __leaf__));
# 33 "/usr/include/netdb.h" 2 3 4
# 1 "/usr/include/bits/netdb.h" 1 3 4
# 26 "/usr/include/bits/netdb.h" 3 4
struct netent
{
  char *n_name;
  char **n_aliases;
  int n_addrtype;
  uint32_t n_net;
};
# 41 "/usr/include/netdb.h" 2 3 4
# 51 "/usr/include/netdb.h" 3 4
extern int *__h_errno_location (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 90 "/usr/include/netdb.h" 3 4
extern void herror (const char *__str) __attribute__ ((__nothrow__ , __leaf__));
extern const char *hstrerror (int __err_num) __attribute__ ((__nothrow__ , __leaf__));
struct hostent
{
  char *h_name;
  char **h_aliases;
  int h_addrtype;
  int h_length;
  char **h_addr_list;
};
extern void sethostent (int __stay_open);
extern void endhostent (void);
extern struct hostent *gethostent (void);
extern struct hostent *gethostbyaddr (const void *__addr, __socklen_t __len,
          int __type);
extern struct hostent *gethostbyname (const char *__name);
# 153 "/usr/include/netdb.h" 3 4
extern struct hostent *gethostbyname2 (const char *__name, int __af);
# 165 "/usr/include/netdb.h" 3 4
extern int gethostent_r (struct hostent *__restrict __result_buf,
    char *__restrict __buf, size_t __buflen,
    struct hostent **__restrict __result,
    int *__restrict __h_errnop);
extern int gethostbyaddr_r (const void *__restrict __addr, __socklen_t __len,
       int __type,
       struct hostent *__restrict __result_buf,
       char *__restrict __buf, size_t __buflen,
       struct hostent **__restrict __result,
       int *__restrict __h_errnop);
extern int gethostbyname_r (const char *__restrict __name,
       struct hostent *__restrict __result_buf,
       char *__restrict __buf, size_t __buflen,
       struct hostent **__restrict __result,
       int *__restrict __h_errnop);
extern int gethostbyname2_r (const char *__restrict __name, int __af,
        struct hostent *__restrict __result_buf,
        char *__restrict __buf, size_t __buflen,
        struct hostent **__restrict __result,
        int *__restrict __h_errnop);
# 196 "/usr/include/netdb.h" 3 4
extern void setnetent (int __stay_open);
extern void endnetent (void);
extern struct netent *getnetent (void);
extern struct netent *getnetbyaddr (uint32_t __net, int __type);
extern struct netent *getnetbyname (const char *__name);
# 235 "/usr/include/netdb.h" 3 4
extern int getnetent_r (struct netent *__restrict __result_buf,
   char *__restrict __buf, size_t __buflen,
   struct netent **__restrict __result,
   int *__restrict __h_errnop);
extern int getnetbyaddr_r (uint32_t __net, int __type,
      struct netent *__restrict __result_buf,
      char *__restrict __buf, size_t __buflen,
      struct netent **__restrict __result,
      int *__restrict __h_errnop);
extern int getnetbyname_r (const char *__restrict __name,
      struct netent *__restrict __result_buf,
      char *__restrict __buf, size_t __buflen,
      struct netent **__restrict __result,
      int *__restrict __h_errnop);
struct servent
{
  char *s_name;
  char **s_aliases;
  int s_port;
  char *s_proto;
};
extern void setservent (int __stay_open);
extern void endservent (void);
extern struct servent *getservent (void);
extern struct servent *getservbyname (const char *__name, const char *__proto);
extern struct servent *getservbyport (int __port, const char *__proto);
# 306 "/usr/include/netdb.h" 3 4
extern int getservent_r (struct servent *__restrict __result_buf,
    char *__restrict __buf, size_t __buflen,
    struct servent **__restrict __result);
extern int getservbyname_r (const char *__restrict __name,
       const char *__restrict __proto,
       struct servent *__restrict __result_buf,
       char *__restrict __buf, size_t __buflen,
       struct servent **__restrict __result);
extern int getservbyport_r (int __port, const char *__restrict __proto,
       struct servent *__restrict __result_buf,
       char *__restrict __buf, size_t __buflen,
       struct servent **__restrict __result);
struct protoent
{
  char *p_name;
  char **p_aliases;
  int p_proto;
};
extern void setprotoent (int __stay_open);
extern void endprotoent (void);
extern struct protoent *getprotoent (void);
extern struct protoent *getprotobyname (const char *__name);
extern struct protoent *getprotobynumber (int __proto);
# 372 "/usr/include/netdb.h" 3 4
extern int getprotoent_r (struct protoent *__restrict __result_buf,
     char *__restrict __buf, size_t __buflen,
     struct protoent **__restrict __result);
extern int getprotobyname_r (const char *__restrict __name,
        struct protoent *__restrict __result_buf,
        char *__restrict __buf, size_t __buflen,
        struct protoent **__restrict __result);
extern int getprotobynumber_r (int __proto,
          struct protoent *__restrict __result_buf,
          char *__restrict __buf, size_t __buflen,
          struct protoent **__restrict __result);
# 393 "/usr/include/netdb.h" 3 4
extern int setnetgrent (const char *__netgroup);
extern void endnetgrent (void);
# 410 "/usr/include/netdb.h" 3 4
extern int getnetgrent (char **__restrict __hostp,
   char **__restrict __userp,
   char **__restrict __domainp);
# 421 "/usr/include/netdb.h" 3 4
extern int innetgr (const char *__netgroup, const char *__host,
      const char *__user, const char *__domain);
extern int getnetgrent_r (char **__restrict __hostp,
     char **__restrict __userp,
     char **__restrict __domainp,
     char *__restrict __buffer, size_t __buflen);
# 449 "/usr/include/netdb.h" 3 4
extern int rcmd (char **__restrict __ahost, unsigned short int __rport,
   const char *__restrict __locuser,
   const char *__restrict __remuser,
   const char *__restrict __cmd, int *__restrict __fd2p);
# 461 "/usr/include/netdb.h" 3 4
extern int rcmd_af (char **__restrict __ahost, unsigned short int __rport,
      const char *__restrict __locuser,
      const char *__restrict __remuser,
      const char *__restrict __cmd, int *__restrict __fd2p,
      sa_family_t __af);
# 477 "/usr/include/netdb.h" 3 4
extern int rexec (char **__restrict __ahost, int __rport,
    const char *__restrict __name,
    const char *__restrict __pass,
    const char *__restrict __cmd, int *__restrict __fd2p);
# 489 "/usr/include/netdb.h" 3 4
extern int rexec_af (char **__restrict __ahost, int __rport,
       const char *__restrict __name,
       const char *__restrict __pass,
       const char *__restrict __cmd, int *__restrict __fd2p,
       sa_family_t __af);
# 503 "/usr/include/netdb.h" 3 4
extern int ruserok (const char *__rhost, int __suser,
      const char *__remuser, const char *__locuser);
# 513 "/usr/include/netdb.h" 3 4
extern int ruserok_af (const char *__rhost, int __suser,
         const char *__remuser, const char *__locuser,
         sa_family_t __af);
# 526 "/usr/include/netdb.h" 3 4
extern int iruserok (uint32_t __raddr, int __suser,
       const char *__remuser, const char *__locuser);
# 537 "/usr/include/netdb.h" 3 4
extern int iruserok_af (const void *__raddr, int __suser,
   const char *__remuser, const char *__locuser,
   sa_family_t __af);
# 549 "/usr/include/netdb.h" 3 4
extern int rresvport (int *__alport);
# 558 "/usr/include/netdb.h" 3 4
extern int rresvport_af (int *__alport, sa_family_t __af);
struct addrinfo
{
  int ai_flags;
  int ai_family;
  int ai_socktype;
  int ai_protocol;
  socklen_t ai_addrlen;
  struct sockaddr *ai_addr;
  char *ai_canonname;
  struct addrinfo *ai_next;
};
# 660 "/usr/include/netdb.h" 3 4
extern int getaddrinfo (const char *__restrict __name,
   const char *__restrict __service,
   const struct addrinfo *__restrict __req,
   struct addrinfo **__restrict __pai);
extern void freeaddrinfo (struct addrinfo *__ai) __attribute__ ((__nothrow__ , __leaf__));
extern const char *gai_strerror (int __ecode) __attribute__ ((__nothrow__ , __leaf__));
extern int getnameinfo (const struct sockaddr *__restrict __sa,
   socklen_t __salen, char *__restrict __host,
   socklen_t __hostlen, char *__restrict __serv,
   socklen_t __servlen, int __flags);
# 711 "/usr/include/netdb.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_NETDB_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: search.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/search.h" 1 3 4
# 22 "/usr/include/search.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/search.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 26 "/usr/include/search.h" 2 3 4
# 44 "/usr/include/search.h" 3 4
extern void insque (void *__elem, void *__prev) __attribute__ ((__nothrow__ , __leaf__));
extern void remque (void *__elem) __attribute__ ((__nothrow__ , __leaf__));
typedef int (*__compar_fn_t) (const void *, const void *);
typedef enum
  {
    FIND,
    ENTER
  }
ACTION;
typedef struct entry
  {
    char *key;
    void *data;
  }
ENTRY;
struct _ENTRY;
# 87 "/usr/include/search.h" 3 4
extern ENTRY *hsearch (ENTRY __item, ACTION __action) __attribute__ ((__nothrow__ , __leaf__));
extern int hcreate (size_t __nel) __attribute__ ((__nothrow__ , __leaf__));
extern void hdestroy (void) __attribute__ ((__nothrow__ , __leaf__));
# 118 "/usr/include/search.h" 3 4
typedef enum
{
  preorder,
  postorder,
  endorder,
  leaf
}
VISIT;
extern void *tsearch (const void *__key, void **__rootp,
        __compar_fn_t __compar);
extern void *tfind (const void *__key, void *const *__rootp,
      __compar_fn_t __compar);
extern void *tdelete (const void *__restrict __key,
        void **__restrict __rootp,
        __compar_fn_t __compar);
typedef void (*__action_fn_t) (const void *__nodep, VISIT __value,
          int __level);
extern void twalk (const void *__root, __action_fn_t __action);
# 164 "/usr/include/search.h" 3 4
extern void *lfind (const void *__key, const void *__base,
      size_t *__nmemb, size_t __size, __compar_fn_t __compar);
extern void *lsearch (const void *__key, void *__base,
        size_t *__nmemb, size_t __size, __compar_fn_t __compar);
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SEARCH_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/procfs.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/procfs.h" 1 3 4
# 30 "/usr/include/sys/procfs.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 31 "/usr/include/sys/procfs.h" 2 3 4
# 1 "/usr/include/sys/time.h" 1 3 4
# 23 "/usr/include/sys/time.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 24 "/usr/include/sys/time.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 25 "/usr/include/sys/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 26 "/usr/include/sys/time.h" 2 3 4
typedef __suseconds_t suseconds_t;
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
# 49 "/usr/include/sys/select.h" 3 4
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 33 "/usr/include/sys/time.h" 2 3 4
# 52 "/usr/include/sys/time.h" 3 4
struct timezone
  {
    int tz_minuteswest;
    int tz_dsttime;
  };
typedef struct timezone *__restrict __timezone_ptr_t;
# 68 "/usr/include/sys/time.h" 3 4
extern int gettimeofday (struct timeval *__restrict __tv,
    __timezone_ptr_t __tz) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int settimeofday (const struct timeval *__tv,
    const struct timezone *__tz)
     __attribute__ ((__nothrow__ , __leaf__));
extern int adjtime (const struct timeval *__delta,
      struct timeval *__olddelta) __attribute__ ((__nothrow__ , __leaf__));
enum __itimer_which
  {
    ITIMER_REAL = 0,
    ITIMER_VIRTUAL = 1,
    ITIMER_PROF = 2
  };
struct itimerval
  {
    struct timeval it_interval;
    struct timeval it_value;
  };
typedef int __itimer_which_t;
extern int getitimer (__itimer_which_t __which,
        struct itimerval *__value) __attribute__ ((__nothrow__ , __leaf__));
extern int setitimer (__itimer_which_t __which,
        const struct itimerval *__restrict __new,
        struct itimerval *__restrict __old) __attribute__ ((__nothrow__ , __leaf__));
extern int utimes (const char *__file, const struct timeval __tvp[2])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int lutimes (const char *__file, const struct timeval __tvp[2])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int futimes (int __fd, const struct timeval __tvp[2]) __attribute__ ((__nothrow__ , __leaf__));
# 186 "/usr/include/sys/time.h" 3 4
# 32 "/usr/include/sys/procfs.h" 2 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 202 "/usr/include/sys/types.h" 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 33 "/usr/include/sys/procfs.h" 2 3 4
# 1 "/usr/include/sys/user.h" 1 3 4
# 27 "/usr/include/sys/user.h" 3 4
struct user_fpregs_struct
{
  unsigned short int cwd;
  unsigned short int swd;
  unsigned short int ftw;
  unsigned short int fop;
  __extension__ unsigned long long int rip;
  __extension__ unsigned long long int rdp;
  unsigned int mxcsr;
  unsigned int mxcr_mask;
  unsigned int st_space[32];
  unsigned int xmm_space[64];
  unsigned int padding[24];
};
struct user_regs_struct
{
  __extension__ unsigned long long int r15;
  __extension__ unsigned long long int r14;
  __extension__ unsigned long long int r13;
  __extension__ unsigned long long int r12;
  __extension__ unsigned long long int rbp;
  __extension__ unsigned long long int rbx;
  __extension__ unsigned long long int r11;
  __extension__ unsigned long long int r10;
  __extension__ unsigned long long int r9;
  __extension__ unsigned long long int r8;
  __extension__ unsigned long long int rax;
  __extension__ unsigned long long int rcx;
  __extension__ unsigned long long int rdx;
  __extension__ unsigned long long int rsi;
  __extension__ unsigned long long int rdi;
  __extension__ unsigned long long int orig_rax;
  __extension__ unsigned long long int rip;
  __extension__ unsigned long long int cs;
  __extension__ unsigned long long int eflags;
  __extension__ unsigned long long int rsp;
  __extension__ unsigned long long int ss;
  __extension__ unsigned long long int fs_base;
  __extension__ unsigned long long int gs_base;
  __extension__ unsigned long long int ds;
  __extension__ unsigned long long int es;
  __extension__ unsigned long long int fs;
  __extension__ unsigned long long int gs;
};
struct user
{
  struct user_regs_struct regs;
  int u_fpvalid;
  struct user_fpregs_struct i387;
  __extension__ unsigned long long int u_tsize;
  __extension__ unsigned long long int u_dsize;
  __extension__ unsigned long long int u_ssize;
  __extension__ unsigned long long int start_code;
  __extension__ unsigned long long int start_stack;
  __extension__ long long int signal;
  int reserved;
  __extension__ union
    {
      struct user_regs_struct* u_ar0;
      __extension__ unsigned long long int __u_ar0_word;
    };
  __extension__ union
    {
      struct user_fpregs_struct* u_fpstate;
      __extension__ unsigned long long int __u_fpstate_word;
    };
  __extension__ unsigned long long int magic;
  char u_comm [32];
  __extension__ unsigned long long int u_debugreg [8];
};
# 34 "/usr/include/sys/procfs.h" 2 3 4
__extension__ typedef unsigned long long elf_greg_t;
# 49 "/usr/include/sys/procfs.h" 3 4
typedef elf_greg_t elf_gregset_t[(sizeof (struct user_regs_struct) / sizeof(elf_greg_t))];
# 63 "/usr/include/sys/procfs.h" 3 4
typedef struct user_fpregs_struct elf_fpregset_t;
struct elf_siginfo
  {
    int si_signo;
    int si_code;
    int si_errno;
  };
# 82 "/usr/include/sys/procfs.h" 3 4
struct elf_prstatus
  {
    struct elf_siginfo pr_info;
    short int pr_cursig;
    unsigned long int pr_sigpend;
    unsigned long int pr_sighold;
    __pid_t pr_pid;
    __pid_t pr_ppid;
    __pid_t pr_pgrp;
    __pid_t pr_sid;
    struct timeval pr_utime;
    struct timeval pr_stime;
    struct timeval pr_cutime;
    struct timeval pr_cstime;
    elf_gregset_t pr_reg;
    int pr_fpvalid;
  };
struct elf_prpsinfo
  {
    char pr_state;
    char pr_sname;
    char pr_zomb;
    char pr_nice;
    unsigned long int pr_flag;
    unsigned int pr_uid;
    unsigned int pr_gid;
    int pr_pid, pr_ppid, pr_pgrp, pr_sid;
    char pr_fname[16];
    char pr_psargs[(80)];
  };
typedef void *psaddr_t;
typedef elf_gregset_t prgregset_t;
typedef elf_fpregset_t prfpregset_t;
typedef __pid_t lwpid_t;
typedef struct elf_prstatus prstatus_t;
typedef struct elf_prpsinfo prpsinfo_t;
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_PROCFS_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: sys/resource.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/resource.h" 1 3 4
# 21 "/usr/include/sys/resource.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/sys/resource.h" 2 3 4
# 1 "/usr/include/bits/resource.h" 1 3 4
# 23 "/usr/include/bits/resource.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 24 "/usr/include/bits/resource.h" 2 3 4
enum __rlimit_resource
{
  RLIMIT_CPU = 0,
  RLIMIT_FSIZE = 1,
  RLIMIT_DATA = 2,
  RLIMIT_STACK = 3,
  RLIMIT_CORE = 4,
  __RLIMIT_RSS = 5,
  RLIMIT_NOFILE = 7,
  __RLIMIT_OFILE = RLIMIT_NOFILE,
  RLIMIT_AS = 9,
  __RLIMIT_NPROC = 6,
  __RLIMIT_MEMLOCK = 8,
  __RLIMIT_LOCKS = 10,
  __RLIMIT_SIGPENDING = 11,
  __RLIMIT_MSGQUEUE = 12,
  __RLIMIT_NICE = 13,
  __RLIMIT_RTPRIO = 14,
  __RLIMIT_RTTIME = 15,
  __RLIMIT_NLIMITS = 16,
  __RLIM_NLIMITS = __RLIMIT_NLIMITS
};
# 131 "/usr/include/bits/resource.h" 3 4
typedef __rlim_t rlim_t;
struct rlimit
  {
    rlim_t rlim_cur;
    rlim_t rlim_max;
  };
# 158 "/usr/include/bits/resource.h" 3 4
enum __rusage_who
{
  RUSAGE_SELF = 0,
  RUSAGE_CHILDREN = -1
# 176 "/usr/include/bits/resource.h" 3 4
};
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 179 "/usr/include/bits/resource.h" 2 3 4
# 1 "/usr/include/bits/types/struct_rusage.h" 1 3 4
# 31 "/usr/include/bits/types/struct_rusage.h" 3 4
struct rusage
  {
    struct timeval ru_utime;
    struct timeval ru_stime;
    __extension__ union
      {
 long int ru_maxrss;
 __syscall_slong_t __ru_maxrss_word;
      };
    __extension__ union
      {
 long int ru_ixrss;
 __syscall_slong_t __ru_ixrss_word;
      };
    __extension__ union
      {
 long int ru_idrss;
 __syscall_slong_t __ru_idrss_word;
      };
    __extension__ union
      {
 long int ru_isrss;
  __syscall_slong_t __ru_isrss_word;
      };
    __extension__ union
      {
 long int ru_minflt;
 __syscall_slong_t __ru_minflt_word;
      };
    __extension__ union
      {
 long int ru_majflt;
 __syscall_slong_t __ru_majflt_word;
      };
    __extension__ union
      {
 long int ru_nswap;
 __syscall_slong_t __ru_nswap_word;
      };
    __extension__ union
      {
 long int ru_inblock;
 __syscall_slong_t __ru_inblock_word;
      };
    __extension__ union
      {
 long int ru_oublock;
 __syscall_slong_t __ru_oublock_word;
      };
    __extension__ union
      {
 long int ru_msgsnd;
 __syscall_slong_t __ru_msgsnd_word;
      };
    __extension__ union
      {
 long int ru_msgrcv;
 __syscall_slong_t __ru_msgrcv_word;
      };
    __extension__ union
      {
 long int ru_nsignals;
 __syscall_slong_t __ru_nsignals_word;
      };
    __extension__ union
      {
 long int ru_nvcsw;
 __syscall_slong_t __ru_nvcsw_word;
      };
    __extension__ union
      {
 long int ru_nivcsw;
 __syscall_slong_t __ru_nivcsw_word;
      };
  };
# 180 "/usr/include/bits/resource.h" 2 3 4
enum __priority_which
{
  PRIO_PROCESS = 0,
  PRIO_PGRP = 1,
  PRIO_USER = 2
};
# 223 "/usr/include/bits/resource.h" 3 4
# 25 "/usr/include/sys/resource.h" 2 3 4
typedef __id_t id_t;
# 42 "/usr/include/sys/resource.h" 3 4
typedef int __rlimit_resource_t;
typedef int __rusage_who_t;
typedef int __priority_which_t;
extern int getrlimit (__rlimit_resource_t __resource,
        struct rlimit *__rlimits) __attribute__ ((__nothrow__ , __leaf__));
# 69 "/usr/include/sys/resource.h" 3 4
extern int setrlimit (__rlimit_resource_t __resource,
        const struct rlimit *__rlimits) __attribute__ ((__nothrow__ , __leaf__));
# 87 "/usr/include/sys/resource.h" 3 4
extern int getrusage (__rusage_who_t __who, struct rusage *__usage) __attribute__ ((__nothrow__ , __leaf__));
extern int getpriority (__priority_which_t __which, id_t __who) __attribute__ ((__nothrow__ , __leaf__));
extern int setpriority (__priority_which_t __which, id_t __who, int __prio)
     __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_RESOURCE_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: unistd.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/unistd.h" 1 3 4
# 25 "/usr/include/unistd.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/unistd.h" 2 3 4
# 202 "/usr/include/unistd.h" 3 4
# 1 "/usr/include/bits/posix_opt.h" 1 3 4
# 203 "/usr/include/unistd.h" 2 3 4
# 1 "/usr/include/bits/environments.h" 1 3 4
# 22 "/usr/include/bits/environments.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/environments.h" 2 3 4
# 207 "/usr/include/unistd.h" 2 3 4
# 217 "/usr/include/unistd.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 218 "/usr/include/unistd.h" 2 3 4
typedef __ssize_t ssize_t;
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 227 "/usr/include/unistd.h" 2 3 4
typedef __gid_t gid_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 255 "/usr/include/unistd.h" 3 4
typedef __useconds_t useconds_t;
typedef __pid_t pid_t;
typedef __intptr_t intptr_t;
typedef __socklen_t socklen_t;
# 287 "/usr/include/unistd.h" 3 4
extern int access (const char *__name, int __type) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 304 "/usr/include/unistd.h" 3 4
extern int faccessat (int __fd, const char *__file, int __type, int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2))) ;
# 334 "/usr/include/unistd.h" 3 4
extern __off_t lseek (int __fd, __off_t __offset, int __whence) __attribute__ ((__nothrow__ , __leaf__));
# 353 "/usr/include/unistd.h" 3 4
extern int close (int __fd);
extern ssize_t read (int __fd, void *__buf, size_t __nbytes) ;
extern ssize_t write (int __fd, const void *__buf, size_t __n) ;
# 376 "/usr/include/unistd.h" 3 4
extern ssize_t pread (int __fd, void *__buf, size_t __nbytes,
        __off_t __offset) ;
extern ssize_t pwrite (int __fd, const void *__buf, size_t __n,
         __off_t __offset) ;
# 417 "/usr/include/unistd.h" 3 4
extern int pipe (int __pipedes[2]) __attribute__ ((__nothrow__ , __leaf__)) ;
# 432 "/usr/include/unistd.h" 3 4
extern unsigned int alarm (unsigned int __seconds) __attribute__ ((__nothrow__ , __leaf__));
# 444 "/usr/include/unistd.h" 3 4
extern unsigned int sleep (unsigned int __seconds);
extern __useconds_t ualarm (__useconds_t __value, __useconds_t __interval)
     __attribute__ ((__nothrow__ , __leaf__));
extern int usleep (__useconds_t __useconds);
# 469 "/usr/include/unistd.h" 3 4
extern int pause (void);
extern int chown (const char *__file, __uid_t __owner, __gid_t __group)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int fchown (int __fd, __uid_t __owner, __gid_t __group) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int lchown (const char *__file, __uid_t __owner, __gid_t __group)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int fchownat (int __fd, const char *__file, __uid_t __owner,
       __gid_t __group, int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2))) ;
extern int chdir (const char *__path) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int fchdir (int __fd) __attribute__ ((__nothrow__ , __leaf__)) ;
# 511 "/usr/include/unistd.h" 3 4
extern char *getcwd (char *__buf, size_t __size) __attribute__ ((__nothrow__ , __leaf__)) ;
# 525 "/usr/include/unistd.h" 3 4
extern char *getwd (char *__buf)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) __attribute__ ((__deprecated__)) ;
extern int dup (int __fd) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int dup2 (int __fd, int __fd2) __attribute__ ((__nothrow__ , __leaf__));
# 543 "/usr/include/unistd.h" 3 4
extern char **__environ;
extern int execve (const char *__path, char *const __argv[],
     char *const __envp[]) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int fexecve (int __fd, char *const __argv[], char *const __envp[])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int execv (const char *__path, char *const __argv[])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int execle (const char *__path, const char *__arg, ...)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int execl (const char *__path, const char *__arg, ...)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int execvp (const char *__file, char *const __argv[])
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int execlp (const char *__file, const char *__arg, ...)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
# 598 "/usr/include/unistd.h" 3 4
extern int nice (int __inc) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void _exit (int __status) __attribute__ ((__noreturn__));
# 1 "/usr/include/bits/confname.h" 1 3 4
# 24 "/usr/include/bits/confname.h" 3 4
enum
  {
    _PC_LINK_MAX,
    _PC_MAX_CANON,
    _PC_MAX_INPUT,
    _PC_NAME_MAX,
    _PC_PATH_MAX,
    _PC_PIPE_BUF,
    _PC_CHOWN_RESTRICTED,
    _PC_NO_TRUNC,
    _PC_VDISABLE,
    _PC_SYNC_IO,
    _PC_ASYNC_IO,
    _PC_PRIO_IO,
    _PC_SOCK_MAXBUF,
    _PC_FILESIZEBITS,
    _PC_REC_INCR_XFER_SIZE,
    _PC_REC_MAX_XFER_SIZE,
    _PC_REC_MIN_XFER_SIZE,
    _PC_REC_XFER_ALIGN,
    _PC_ALLOC_SIZE_MIN,
    _PC_SYMLINK_MAX,
    _PC_2_SYMLINKS
  };
enum
  {
    _SC_ARG_MAX,
    _SC_CHILD_MAX,
    _SC_CLK_TCK,
    _SC_NGROUPS_MAX,
    _SC_OPEN_MAX,
    _SC_STREAM_MAX,
    _SC_TZNAME_MAX,
    _SC_JOB_CONTROL,
    _SC_SAVED_IDS,
    _SC_REALTIME_SIGNALS,
    _SC_PRIORITY_SCHEDULING,
    _SC_TIMERS,
    _SC_ASYNCHRONOUS_IO,
    _SC_PRIORITIZED_IO,
    _SC_SYNCHRONIZED_IO,
    _SC_FSYNC,
    _SC_MAPPED_FILES,
    _SC_MEMLOCK,
    _SC_MEMLOCK_RANGE,
    _SC_MEMORY_PROTECTION,
    _SC_MESSAGE_PASSING,
    _SC_SEMAPHORES,
    _SC_SHARED_MEMORY_OBJECTS,
    _SC_AIO_LISTIO_MAX,
    _SC_AIO_MAX,
    _SC_AIO_PRIO_DELTA_MAX,
    _SC_DELAYTIMER_MAX,
    _SC_MQ_OPEN_MAX,
    _SC_MQ_PRIO_MAX,
    _SC_VERSION,
    _SC_PAGESIZE,
    _SC_RTSIG_MAX,
    _SC_SEM_NSEMS_MAX,
    _SC_SEM_VALUE_MAX,
    _SC_SIGQUEUE_MAX,
    _SC_TIMER_MAX,
    _SC_BC_BASE_MAX,
    _SC_BC_DIM_MAX,
    _SC_BC_SCALE_MAX,
    _SC_BC_STRING_MAX,
    _SC_COLL_WEIGHTS_MAX,
    _SC_EQUIV_CLASS_MAX,
    _SC_EXPR_NEST_MAX,
    _SC_LINE_MAX,
    _SC_RE_DUP_MAX,
    _SC_CHARCLASS_NAME_MAX,
    _SC_2_VERSION,
    _SC_2_C_BIND,
    _SC_2_C_DEV,
    _SC_2_FORT_DEV,
    _SC_2_FORT_RUN,
    _SC_2_SW_DEV,
    _SC_2_LOCALEDEF,
    _SC_PII,
    _SC_PII_XTI,
    _SC_PII_SOCKET,
    _SC_PII_INTERNET,
    _SC_PII_OSI,
    _SC_POLL,
    _SC_SELECT,
    _SC_UIO_MAXIOV,
    _SC_IOV_MAX = _SC_UIO_MAXIOV,
    _SC_PII_INTERNET_STREAM,
    _SC_PII_INTERNET_DGRAM,
    _SC_PII_OSI_COTS,
    _SC_PII_OSI_CLTS,
    _SC_PII_OSI_M,
    _SC_T_IOV_MAX,
    _SC_THREADS,
    _SC_THREAD_SAFE_FUNCTIONS,
    _SC_GETGR_R_SIZE_MAX,
    _SC_GETPW_R_SIZE_MAX,
    _SC_LOGIN_NAME_MAX,
    _SC_TTY_NAME_MAX,
    _SC_THREAD_DESTRUCTOR_ITERATIONS,
    _SC_THREAD_KEYS_MAX,
    _SC_THREAD_STACK_MIN,
    _SC_THREAD_THREADS_MAX,
    _SC_THREAD_ATTR_STACKADDR,
    _SC_THREAD_ATTR_STACKSIZE,
    _SC_THREAD_PRIORITY_SCHEDULING,
    _SC_THREAD_PRIO_INHERIT,
    _SC_THREAD_PRIO_PROTECT,
    _SC_THREAD_PROCESS_SHARED,
    _SC_NPROCESSORS_CONF,
    _SC_NPROCESSORS_ONLN,
    _SC_PHYS_PAGES,
    _SC_AVPHYS_PAGES,
    _SC_ATEXIT_MAX,
    _SC_PASS_MAX,
    _SC_XOPEN_VERSION,
    _SC_XOPEN_XCU_VERSION,
    _SC_XOPEN_UNIX,
    _SC_XOPEN_CRYPT,
    _SC_XOPEN_ENH_I18N,
    _SC_XOPEN_SHM,
    _SC_2_CHAR_TERM,
    _SC_2_C_VERSION,
    _SC_2_UPE,
    _SC_XOPEN_XPG2,
    _SC_XOPEN_XPG3,
    _SC_XOPEN_XPG4,
    _SC_CHAR_BIT,
    _SC_CHAR_MAX,
    _SC_CHAR_MIN,
    _SC_INT_MAX,
    _SC_INT_MIN,
    _SC_LONG_BIT,
    _SC_WORD_BIT,
    _SC_MB_LEN_MAX,
    _SC_NZERO,
    _SC_SSIZE_MAX,
    _SC_SCHAR_MAX,
    _SC_SCHAR_MIN,
    _SC_SHRT_MAX,
    _SC_SHRT_MIN,
    _SC_UCHAR_MAX,
    _SC_UINT_MAX,
    _SC_ULONG_MAX,
    _SC_USHRT_MAX,
    _SC_NL_ARGMAX,
    _SC_NL_LANGMAX,
    _SC_NL_MSGMAX,
    _SC_NL_NMAX,
    _SC_NL_SETMAX,
    _SC_NL_TEXTMAX,
    _SC_XBS5_ILP32_OFF32,
    _SC_XBS5_ILP32_OFFBIG,
    _SC_XBS5_LP64_OFF64,
    _SC_XBS5_LPBIG_OFFBIG,
    _SC_XOPEN_LEGACY,
    _SC_XOPEN_REALTIME,
    _SC_XOPEN_REALTIME_THREADS,
    _SC_ADVISORY_INFO,
    _SC_BARRIERS,
    _SC_BASE,
    _SC_C_LANG_SUPPORT,
    _SC_C_LANG_SUPPORT_R,
    _SC_CLOCK_SELECTION,
    _SC_CPUTIME,
    _SC_THREAD_CPUTIME,
    _SC_DEVICE_IO,
    _SC_DEVICE_SPECIFIC,
    _SC_DEVICE_SPECIFIC_R,
    _SC_FD_MGMT,
    _SC_FIFO,
    _SC_PIPE,
    _SC_FILE_ATTRIBUTES,
    _SC_FILE_LOCKING,
    _SC_FILE_SYSTEM,
    _SC_MONOTONIC_CLOCK,
    _SC_MULTI_PROCESS,
    _SC_SINGLE_PROCESS,
    _SC_NETWORKING,
    _SC_READER_WRITER_LOCKS,
    _SC_SPIN_LOCKS,
    _SC_REGEXP,
    _SC_REGEX_VERSION,
    _SC_SHELL,
    _SC_SIGNALS,
    _SC_SPAWN,
    _SC_SPORADIC_SERVER,
    _SC_THREAD_SPORADIC_SERVER,
    _SC_SYSTEM_DATABASE,
    _SC_SYSTEM_DATABASE_R,
    _SC_TIMEOUTS,
    _SC_TYPED_MEMORY_OBJECTS,
    _SC_USER_GROUPS,
    _SC_USER_GROUPS_R,
    _SC_2_PBS,
    _SC_2_PBS_ACCOUNTING,
    _SC_2_PBS_LOCATE,
    _SC_2_PBS_MESSAGE,
    _SC_2_PBS_TRACK,
    _SC_SYMLOOP_MAX,
    _SC_STREAMS,
    _SC_2_PBS_CHECKPOINT,
    _SC_V6_ILP32_OFF32,
    _SC_V6_ILP32_OFFBIG,
    _SC_V6_LP64_OFF64,
    _SC_V6_LPBIG_OFFBIG,
    _SC_HOST_NAME_MAX,
    _SC_TRACE,
    _SC_TRACE_EVENT_FILTER,
    _SC_TRACE_INHERIT,
    _SC_TRACE_LOG,
    _SC_LEVEL1_ICACHE_SIZE,
    _SC_LEVEL1_ICACHE_ASSOC,
    _SC_LEVEL1_ICACHE_LINESIZE,
    _SC_LEVEL1_DCACHE_SIZE,
    _SC_LEVEL1_DCACHE_ASSOC,
    _SC_LEVEL1_DCACHE_LINESIZE,
    _SC_LEVEL2_CACHE_SIZE,
    _SC_LEVEL2_CACHE_ASSOC,
    _SC_LEVEL2_CACHE_LINESIZE,
    _SC_LEVEL3_CACHE_SIZE,
    _SC_LEVEL3_CACHE_ASSOC,
    _SC_LEVEL3_CACHE_LINESIZE,
    _SC_LEVEL4_CACHE_SIZE,
    _SC_LEVEL4_CACHE_ASSOC,
    _SC_LEVEL4_CACHE_LINESIZE,
    _SC_IPV6 = _SC_LEVEL1_ICACHE_SIZE + 50,
    _SC_RAW_SOCKETS,
    _SC_V7_ILP32_OFF32,
    _SC_V7_ILP32_OFFBIG,
    _SC_V7_LP64_OFF64,
    _SC_V7_LPBIG_OFFBIG,
    _SC_SS_REPL_MAX,
    _SC_TRACE_EVENT_NAME_MAX,
    _SC_TRACE_NAME_MAX,
    _SC_TRACE_SYS_MAX,
    _SC_TRACE_USER_EVENT_MAX,
    _SC_XOPEN_STREAMS,
    _SC_THREAD_ROBUST_PRIO_INHERIT,
    _SC_THREAD_ROBUST_PRIO_PROTECT
  };
enum
  {
    _CS_PATH,
    _CS_V6_WIDTH_RESTRICTED_ENVS,
    _CS_GNU_LIBC_VERSION,
    _CS_GNU_LIBPTHREAD_VERSION,
    _CS_V5_WIDTH_RESTRICTED_ENVS,
    _CS_V7_WIDTH_RESTRICTED_ENVS,
    _CS_LFS_CFLAGS = 1000,
    _CS_LFS_LDFLAGS,
    _CS_LFS_LIBS,
    _CS_LFS_LINTFLAGS,
    _CS_LFS64_CFLAGS,
    _CS_LFS64_LDFLAGS,
    _CS_LFS64_LIBS,
    _CS_LFS64_LINTFLAGS,
    _CS_XBS5_ILP32_OFF32_CFLAGS = 1100,
    _CS_XBS5_ILP32_OFF32_LDFLAGS,
    _CS_XBS5_ILP32_OFF32_LIBS,
    _CS_XBS5_ILP32_OFF32_LINTFLAGS,
    _CS_XBS5_ILP32_OFFBIG_CFLAGS,
    _CS_XBS5_ILP32_OFFBIG_LDFLAGS,
    _CS_XBS5_ILP32_OFFBIG_LIBS,
    _CS_XBS5_ILP32_OFFBIG_LINTFLAGS,
    _CS_XBS5_LP64_OFF64_CFLAGS,
    _CS_XBS5_LP64_OFF64_LDFLAGS,
    _CS_XBS5_LP64_OFF64_LIBS,
    _CS_XBS5_LP64_OFF64_LINTFLAGS,
    _CS_XBS5_LPBIG_OFFBIG_CFLAGS,
    _CS_XBS5_LPBIG_OFFBIG_LDFLAGS,
    _CS_XBS5_LPBIG_OFFBIG_LIBS,
    _CS_XBS5_LPBIG_OFFBIG_LINTFLAGS,
    _CS_POSIX_V6_ILP32_OFF32_CFLAGS,
    _CS_POSIX_V6_ILP32_OFF32_LDFLAGS,
    _CS_POSIX_V6_ILP32_OFF32_LIBS,
    _CS_POSIX_V6_ILP32_OFF32_LINTFLAGS,
    _CS_POSIX_V6_ILP32_OFFBIG_CFLAGS,
    _CS_POSIX_V6_ILP32_OFFBIG_LDFLAGS,
    _CS_POSIX_V6_ILP32_OFFBIG_LIBS,
    _CS_POSIX_V6_ILP32_OFFBIG_LINTFLAGS,
    _CS_POSIX_V6_LP64_OFF64_CFLAGS,
    _CS_POSIX_V6_LP64_OFF64_LDFLAGS,
    _CS_POSIX_V6_LP64_OFF64_LIBS,
    _CS_POSIX_V6_LP64_OFF64_LINTFLAGS,
    _CS_POSIX_V6_LPBIG_OFFBIG_CFLAGS,
    _CS_POSIX_V6_LPBIG_OFFBIG_LDFLAGS,
    _CS_POSIX_V6_LPBIG_OFFBIG_LIBS,
    _CS_POSIX_V6_LPBIG_OFFBIG_LINTFLAGS,
    _CS_POSIX_V7_ILP32_OFF32_CFLAGS,
    _CS_POSIX_V7_ILP32_OFF32_LDFLAGS,
    _CS_POSIX_V7_ILP32_OFF32_LIBS,
    _CS_POSIX_V7_ILP32_OFF32_LINTFLAGS,
    _CS_POSIX_V7_ILP32_OFFBIG_CFLAGS,
    _CS_POSIX_V7_ILP32_OFFBIG_LDFLAGS,
    _CS_POSIX_V7_ILP32_OFFBIG_LIBS,
    _CS_POSIX_V7_ILP32_OFFBIG_LINTFLAGS,
    _CS_POSIX_V7_LP64_OFF64_CFLAGS,
    _CS_POSIX_V7_LP64_OFF64_LDFLAGS,
    _CS_POSIX_V7_LP64_OFF64_LIBS,
    _CS_POSIX_V7_LP64_OFF64_LINTFLAGS,
    _CS_POSIX_V7_LPBIG_OFFBIG_CFLAGS,
    _CS_POSIX_V7_LPBIG_OFFBIG_LDFLAGS,
    _CS_POSIX_V7_LPBIG_OFFBIG_LIBS,
    _CS_POSIX_V7_LPBIG_OFFBIG_LINTFLAGS,
    _CS_V6_ENV,
    _CS_V7_ENV
  };
# 610 "/usr/include/unistd.h" 2 3 4
extern long int pathconf (const char *__path, int __name)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern long int fpathconf (int __fd, int __name) __attribute__ ((__nothrow__ , __leaf__));
extern long int sysconf (int __name) __attribute__ ((__nothrow__ , __leaf__));
extern size_t confstr (int __name, char *__buf, size_t __len) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t getpid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t getppid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t getpgrp (void) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t __getpgid (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t getpgid (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern int setpgid (__pid_t __pid, __pid_t __pgid) __attribute__ ((__nothrow__ , __leaf__));
# 660 "/usr/include/unistd.h" 3 4
extern int setpgrp (void) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t setsid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __pid_t getsid (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern __uid_t getuid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __uid_t geteuid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __gid_t getgid (void) __attribute__ ((__nothrow__ , __leaf__));
extern __gid_t getegid (void) __attribute__ ((__nothrow__ , __leaf__));
extern int getgroups (int __size, __gid_t __list[]) __attribute__ ((__nothrow__ , __leaf__)) ;
# 700 "/usr/include/unistd.h" 3 4
extern int setuid (__uid_t __uid) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int setreuid (__uid_t __ruid, __uid_t __euid) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int seteuid (__uid_t __uid) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int setgid (__gid_t __gid) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int setregid (__gid_t __rgid, __gid_t __egid) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int setegid (__gid_t __gid) __attribute__ ((__nothrow__ , __leaf__)) ;
# 756 "/usr/include/unistd.h" 3 4
extern __pid_t fork (void) __attribute__ ((__nothrow__));
extern __pid_t vfork (void) __attribute__ ((__nothrow__ , __leaf__));
extern char *ttyname (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int ttyname_r (int __fd, char *__buf, size_t __buflen)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2))) ;
extern int isatty (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int ttyslot (void) __attribute__ ((__nothrow__ , __leaf__));
extern int link (const char *__from, const char *__to)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2))) ;
extern int linkat (int __fromfd, const char *__from, int __tofd,
     const char *__to, int __flags)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 4))) ;
extern int symlink (const char *__from, const char *__to)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2))) ;
extern ssize_t readlink (const char *__restrict __path,
    char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2))) ;
extern int symlinkat (const char *__from, int __tofd,
        const char *__to) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 3))) ;
extern ssize_t readlinkat (int __fd, const char *__restrict __path,
      char *__restrict __buf, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3))) ;
extern int unlink (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int unlinkat (int __fd, const char *__name, int __flag)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
extern int rmdir (const char *__path) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern __pid_t tcgetpgrp (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int tcsetpgrp (int __fd, __pid_t __pgrp_id) __attribute__ ((__nothrow__ , __leaf__));
extern char *getlogin (void);
extern int getlogin_r (char *__name, size_t __name_len) __attribute__ ((__nonnull__ (1)));
extern int setlogin (const char *__name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1 "/usr/include/bits/getopt_posix.h" 1 3 4
# 27 "/usr/include/bits/getopt_posix.h" 3 4
# 1 "/usr/include/bits/getopt_core.h" 1 3 4
# 28 "/usr/include/bits/getopt_core.h" 3 4
extern char *optarg;
# 50 "/usr/include/bits/getopt_core.h" 3 4
extern int optind;
extern int opterr;
extern int optopt;
# 91 "/usr/include/bits/getopt_core.h" 3 4
extern int getopt (int ___argc, char *const *___argv, const char *__shortopts)
       __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
# 28 "/usr/include/bits/getopt_posix.h" 2 3 4
# 49 "/usr/include/bits/getopt_posix.h" 3 4
# 870 "/usr/include/unistd.h" 2 3 4
extern int gethostname (char *__name, size_t __len) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int sethostname (const char *__name, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int sethostid (long int __id) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int getdomainname (char *__name, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int setdomainname (const char *__name, size_t __len)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int vhangup (void) __attribute__ ((__nothrow__ , __leaf__));
extern int revoke (const char *__file) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern int profil (unsigned short int *__sample_buffer, size_t __size,
     size_t __offset, unsigned int __scale)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int acct (const char *__name) __attribute__ ((__nothrow__ , __leaf__));
extern char *getusershell (void) __attribute__ ((__nothrow__ , __leaf__));
extern void endusershell (void) __attribute__ ((__nothrow__ , __leaf__));
extern void setusershell (void) __attribute__ ((__nothrow__ , __leaf__));
extern int daemon (int __nochdir, int __noclose) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int chroot (const char *__path) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
extern char *getpass (const char *__prompt) __attribute__ ((__nonnull__ (1)));
extern int fsync (int __fd);
# 967 "/usr/include/unistd.h" 3 4
extern long int gethostid (void);
extern void sync (void) __attribute__ ((__nothrow__ , __leaf__));
extern int getpagesize (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int getdtablesize (void) __attribute__ ((__nothrow__ , __leaf__));
# 991 "/usr/include/unistd.h" 3 4
extern int truncate (const char *__file, __off_t __length)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) ;
# 1014 "/usr/include/unistd.h" 3 4
extern int ftruncate (int __fd, __off_t __length) __attribute__ ((__nothrow__ , __leaf__)) ;
# 1035 "/usr/include/unistd.h" 3 4
extern int brk (void *__addr) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void *sbrk (intptr_t __delta) __attribute__ ((__nothrow__ , __leaf__));
# 1056 "/usr/include/unistd.h" 3 4
extern long int syscall (long int __sysno, ...) __attribute__ ((__nothrow__ , __leaf__));
# 1079 "/usr/include/unistd.h" 3 4
extern int lockf (int __fd, int __cmd, __off_t __len) ;
# 1115 "/usr/include/unistd.h" 3 4
extern int fdatasync (int __fildes);
# 1124 "/usr/include/unistd.h" 3 4
extern char *crypt (const char *__key, const char *__salt)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
# 1161 "/usr/include/unistd.h" 3 4
int getentropy (void *__buffer, size_t __length) ;
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_UNISTD_H" to "1"
================================================================================
TEST check from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:77)
TESTING: check from config.headers(config/BuildSystem/config/headers.py:77)
  Checks for "header", and defines HAVE_"header" if found
Checking for header: netinet/in.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/netinet/in.h" 1 3 4
# 21 "/usr/include/netinet/in.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/netinet/in.h" 2 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 22 "/usr/include/bits/stdint-uintn.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 23 "/usr/include/bits/stdint-uintn.h" 2 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 23 "/usr/include/netinet/in.h" 2 3 4
# 1 "/usr/include/sys/socket.h" 1 3 4
# 24 "/usr/include/sys/socket.h" 3 4
# 1 "/usr/include/bits/types/struct_iovec.h" 1 3 4
# 23 "/usr/include/bits/types/struct_iovec.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 24 "/usr/include/bits/types/struct_iovec.h" 2 3 4
struct iovec
  {
    void *iov_base;
    size_t iov_len;
  };
# 27 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 29 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/socket.h" 1 3 4
# 27 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 28 "/usr/include/bits/socket.h" 2 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 27 "/usr/include/sys/types.h" 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 30 "/usr/include/bits/socket.h" 2 3 4
typedef __socklen_t socklen_t;
# 1 "/usr/include/bits/socket_type.h" 1 3 4
# 24 "/usr/include/bits/socket_type.h" 3 4
enum __socket_type
{
  SOCK_STREAM = 1,
  SOCK_DGRAM = 2,
  SOCK_RAW = 3,
  SOCK_RDM = 4,
  SOCK_SEQPACKET = 5,
  SOCK_DCCP = 6,
  SOCK_PACKET = 10,
  SOCK_CLOEXEC = 02000000,
  SOCK_NONBLOCK = 00004000
};
# 39 "/usr/include/bits/socket.h" 2 3 4
# 172 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/bits/sockaddr.h" 1 3 4
# 28 "/usr/include/bits/sockaddr.h" 3 4
typedef unsigned short int sa_family_t;
# 173 "/usr/include/bits/socket.h" 2 3 4
struct sockaddr
  {
    sa_family_t sa_family;
    char sa_data[14];
  };
# 188 "/usr/include/bits/socket.h" 3 4
struct sockaddr_storage
  {
    sa_family_t ss_family;
    char __ss_padding[(128 - (sizeof (unsigned short int)) - sizeof (unsigned long int))];
    unsigned long int __ss_align;
  };
enum
  {
    MSG_OOB = 0x01,
    MSG_PEEK = 0x02,
    MSG_DONTROUTE = 0x04,
    MSG_CTRUNC = 0x08,
    MSG_PROXY = 0x10,
    MSG_TRUNC = 0x20,
    MSG_DONTWAIT = 0x40,
    MSG_EOR = 0x80,
    MSG_WAITALL = 0x100,
    MSG_FIN = 0x200,
    MSG_SYN = 0x400,
    MSG_CONFIRM = 0x800,
    MSG_RST = 0x1000,
    MSG_ERRQUEUE = 0x2000,
    MSG_NOSIGNAL = 0x4000,
    MSG_MORE = 0x8000,
    MSG_WAITFORONE = 0x10000,
    MSG_BATCH = 0x40000,
    MSG_ZEROCOPY = 0x4000000,
    MSG_FASTOPEN = 0x20000000,
    MSG_CMSG_CLOEXEC = 0x40000000
  };
struct msghdr
  {
    void *msg_name;
    socklen_t msg_namelen;
    struct iovec *msg_iov;
    size_t msg_iovlen;
    void *msg_control;
    size_t msg_controllen;
    int msg_flags;
  };
struct cmsghdr
  {
    size_t cmsg_len;
    int cmsg_level;
    int cmsg_type;
    __extension__ unsigned char __cmsg_data [];
  };
# 302 "/usr/include/bits/socket.h" 3 4
extern struct cmsghdr *__cmsg_nxthdr (struct msghdr *__mhdr,
          struct cmsghdr *__cmsg) __attribute__ ((__nothrow__ , __leaf__));
# 329 "/usr/include/bits/socket.h" 3 4
enum
  {
    SCM_RIGHTS = 0x01
  };
# 390 "/usr/include/bits/socket.h" 3 4
# 1 "/usr/include/asm/socket.h" 1 3 4
# 1 "/usr/include/asm-generic/socket.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 1 3 4
# 1 "/usr/include/asm-generic/sockios.h" 1 3 4
# 1 "/usr/include/asm/sockios.h" 2 3 4
# 6 "/usr/include/asm-generic/socket.h" 2 3 4
# 1 "/usr/include/asm/socket.h" 2 3 4
# 391 "/usr/include/bits/socket.h" 2 3 4
# 444 "/usr/include/bits/socket.h" 3 4
struct linger
  {
    int l_onoff;
    int l_linger;
  };
# 34 "/usr/include/sys/socket.h" 2 3 4
# 1 "/usr/include/bits/types/struct_osockaddr.h" 1 3 4
struct osockaddr
{
  unsigned short int sa_family;
  unsigned char sa_data[14];
};
# 37 "/usr/include/sys/socket.h" 2 3 4
enum
{
  SHUT_RD = 0,
  SHUT_WR,
  SHUT_RDWR
};
# 102 "/usr/include/sys/socket.h" 3 4
extern int socket (int __domain, int __type, int __protocol) __attribute__ ((__nothrow__ , __leaf__));
extern int socketpair (int __domain, int __type, int __protocol,
         int __fds[2]) __attribute__ ((__nothrow__ , __leaf__));
extern int bind (int __fd, const struct sockaddr * __addr, socklen_t __len)
     __attribute__ ((__nothrow__ , __leaf__));
extern int getsockname (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
# 126 "/usr/include/sys/socket.h" 3 4
extern int connect (int __fd, const struct sockaddr * __addr, socklen_t __len);
extern int getpeername (int __fd, struct sockaddr *__restrict __addr,
   socklen_t *__restrict __len) __attribute__ ((__nothrow__ , __leaf__));
extern ssize_t send (int __fd, const void *__buf, size_t __n, int __flags);
extern ssize_t recv (int __fd, void *__buf, size_t __n, int __flags);
extern ssize_t sendto (int __fd, const void *__buf, size_t __n,
         int __flags, const struct sockaddr * __addr,
         socklen_t __addr_len);
# 163 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvfrom (int __fd, void *__restrict __buf, size_t __n,
    int __flags, struct sockaddr *__restrict __addr,
    socklen_t *__restrict __addr_len);
extern ssize_t sendmsg (int __fd, const struct msghdr *__message,
   int __flags);
# 191 "/usr/include/sys/socket.h" 3 4
extern ssize_t recvmsg (int __fd, struct msghdr *__message, int __flags);
# 208 "/usr/include/sys/socket.h" 3 4
extern int getsockopt (int __fd, int __level, int __optname,
         void *__restrict __optval,
         socklen_t *__restrict __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int setsockopt (int __fd, int __level, int __optname,
         const void *__optval, socklen_t __optlen) __attribute__ ((__nothrow__ , __leaf__));
extern int listen (int __fd, int __n) __attribute__ ((__nothrow__ , __leaf__));
# 232 "/usr/include/sys/socket.h" 3 4
extern int accept (int __fd, struct sockaddr *__restrict __addr,
     socklen_t *__restrict __addr_len);
# 250 "/usr/include/sys/socket.h" 3 4
extern int shutdown (int __fd, int __how) __attribute__ ((__nothrow__ , __leaf__));
extern int sockatmark (int __fd) __attribute__ ((__nothrow__ , __leaf__));
extern int isfdtype (int __fd, int __fdtype) __attribute__ ((__nothrow__ , __leaf__));
# 272 "/usr/include/sys/socket.h" 3 4
# 24 "/usr/include/netinet/in.h" 2 3 4
typedef uint32_t in_addr_t;
struct in_addr
  {
    in_addr_t s_addr;
  };
# 1 "/usr/include/bits/in.h" 1 3 4
# 142 "/usr/include/bits/in.h" 3 4
struct ip_opts
  {
    struct in_addr ip_dst;
    char ip_opts[40];
  };
struct ip_mreqn
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_address;
    int imr_ifindex;
  };
struct in_pktinfo
  {
    int ipi_ifindex;
    struct in_addr ipi_spec_dst;
    struct in_addr ipi_addr;
  };
# 38 "/usr/include/netinet/in.h" 2 3 4
enum
  {
    IPPROTO_IP = 0,
    IPPROTO_ICMP = 1,
    IPPROTO_IGMP = 2,
    IPPROTO_IPIP = 4,
    IPPROTO_TCP = 6,
    IPPROTO_EGP = 8,
    IPPROTO_PUP = 12,
    IPPROTO_UDP = 17,
    IPPROTO_IDP = 22,
    IPPROTO_TP = 29,
    IPPROTO_DCCP = 33,
    IPPROTO_IPV6 = 41,
    IPPROTO_RSVP = 46,
    IPPROTO_GRE = 47,
    IPPROTO_ESP = 50,
    IPPROTO_AH = 51,
    IPPROTO_MTP = 92,
    IPPROTO_BEETPH = 94,
    IPPROTO_ENCAP = 98,
    IPPROTO_PIM = 103,
    IPPROTO_COMP = 108,
    IPPROTO_SCTP = 132,
    IPPROTO_UDPLITE = 136,
    IPPROTO_MPLS = 137,
    IPPROTO_RAW = 255,
    IPPROTO_MAX
  };
enum
  {
    IPPROTO_HOPOPTS = 0,
    IPPROTO_ROUTING = 43,
    IPPROTO_FRAGMENT = 44,
    IPPROTO_ICMPV6 = 58,
    IPPROTO_NONE = 59,
    IPPROTO_DSTOPTS = 60,
    IPPROTO_MH = 135
  };
typedef uint16_t in_port_t;
enum
  {
    IPPORT_ECHO = 7,
    IPPORT_DISCARD = 9,
    IPPORT_SYSTAT = 11,
    IPPORT_DAYTIME = 13,
    IPPORT_NETSTAT = 15,
    IPPORT_FTP = 21,
    IPPORT_TELNET = 23,
    IPPORT_SMTP = 25,
    IPPORT_TIMESERVER = 37,
    IPPORT_NAMESERVER = 42,
    IPPORT_WHOIS = 43,
    IPPORT_MTP = 57,
    IPPORT_TFTP = 69,
    IPPORT_RJE = 77,
    IPPORT_FINGER = 79,
    IPPORT_TTYLINK = 87,
    IPPORT_SUPDUP = 95,
    IPPORT_EXECSERVER = 512,
    IPPORT_LOGINSERVER = 513,
    IPPORT_CMDSERVER = 514,
    IPPORT_EFSSERVER = 520,
    IPPORT_BIFFUDP = 512,
    IPPORT_WHOSERVER = 513,
    IPPORT_ROUTESERVER = 520,
    IPPORT_RESERVED = 1024,
    IPPORT_USERRESERVED = 5000
  };
# 211 "/usr/include/netinet/in.h" 3 4
struct in6_addr
  {
    union
      {
 uint8_t __u6_addr8[16];
 uint16_t __u6_addr16[8];
 uint32_t __u6_addr32[4];
      } __in6_u;
  };
extern const struct in6_addr in6addr_any;
extern const struct in6_addr in6addr_loopback;
# 237 "/usr/include/netinet/in.h" 3 4
struct sockaddr_in
  {
    sa_family_t sin_family;
    in_port_t sin_port;
    struct in_addr sin_addr;
    unsigned char sin_zero[sizeof (struct sockaddr) -
      (sizeof (unsigned short int)) -
      sizeof (in_port_t) -
      sizeof (struct in_addr)];
  };
struct sockaddr_in6
  {
    sa_family_t sin6_family;
    in_port_t sin6_port;
    uint32_t sin6_flowinfo;
    struct in6_addr sin6_addr;
    uint32_t sin6_scope_id;
  };
struct ip_mreq
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_interface;
  };
struct ip_mreq_source
  {
    struct in_addr imr_multiaddr;
    struct in_addr imr_interface;
    struct in_addr imr_sourceaddr;
  };
struct ipv6_mreq
  {
    struct in6_addr ipv6mr_multiaddr;
    unsigned int ipv6mr_interface;
  };
struct group_req
  {
    uint32_t gr_interface;
    struct sockaddr_storage gr_group;
  };
struct group_source_req
  {
    uint32_t gsr_interface;
    struct sockaddr_storage gsr_group;
    struct sockaddr_storage gsr_source;
  };
struct ip_msfilter
  {
    struct in_addr imsf_multiaddr;
    struct in_addr imsf_interface;
    uint32_t imsf_fmode;
    uint32_t imsf_numsrc;
    struct in_addr imsf_slist[1];
  };
struct group_filter
  {
    uint32_t gf_interface;
    struct sockaddr_storage gf_group;
    uint32_t gf_fmode;
    uint32_t gf_numsrc;
    struct sockaddr_storage gf_slist[1];
};
# 374 "/usr/include/netinet/in.h" 3 4
extern uint32_t ntohl (uint32_t __netlong) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint16_t ntohs (uint16_t __netshort)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint32_t htonl (uint32_t __hostlong)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern uint16_t htons (uint16_t __hostshort)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 386 "/usr/include/netinet/in.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 387 "/usr/include/netinet/in.h" 2 3 4
# 502 "/usr/include/netinet/in.h" 3 4
extern int bindresvport (int __sockfd, struct sockaddr_in *__sock_in) __attribute__ ((__nothrow__ , __leaf__));
extern int bindresvport6 (int __sockfd, struct sockaddr_in6 *__sock_in)
     __attribute__ ((__nothrow__ , __leaf__));
# 630 "/usr/include/netinet/in.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_NETINET_IN_H" to "1"
================================================================================
TEST checkRecursiveMacros from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:223)
TESTING: checkRecursiveMacros from config.headers(config/BuildSystem/config/headers.py:223)
  Checks that the preprocessor allows recursive macros, and if not defines HAVE_BROKEN_RECURSIVE_MACRO
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.headers/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.headers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
void a(int i, int j) {}
#define a(b) a(b,__LINE__)

int main() {
a(0);
;
  return 0;
}
================================================================================
TEST configureCacheDetails from config.utilities.cacheDetails(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/cacheDetails.py:78)
TESTING: configureCacheDetails from config.utilities.cacheDetails(config/BuildSystem/config/utilities/cacheDetails.py:78)
  Try to determine the size and associativity of the cache.
                Pushing language C
                          All intermediate test results are stored in /tmp/petsc-VuReVk/config.utilities.cacheDetails
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.headers -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <unistd.h>
long getconf_LEVEL1_DCACHE_SIZE() { long val = sysconf(_SC_LEVEL1_DCACHE_SIZE); return (16 <= val && val <= 2147483647) ? val : 32768; }

int main() {
;
  return 0;
}
                Popping language C
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <unistd.h>
long getconf_LEVEL1_DCACHE_SIZE() { long val = sysconf(_SC_LEVEL1_DCACHE_SIZE); return (16 <= val && val <= 2147483647) ? val : 32768; }

int main() {
FILE *output = fopen("conftestval","w"); if (!output) return 1; fprintf(output,"%ld",getconf_LEVEL1_DCACHE_SIZE()); fclose(output);;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
              Popping language C
              Defined "LEVEL1_DCACHE_SIZE" to "32768"
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <unistd.h>
long getconf_LEVEL1_DCACHE_LINESIZE() { long val = sysconf(_SC_LEVEL1_DCACHE_LINESIZE); return (16 <= val && val <= 2147483647) ? val : 32; }

int main() {
FILE *output = fopen("conftestval","w"); if (!output) return 1; fprintf(output,"%ld",getconf_LEVEL1_DCACHE_LINESIZE()); fclose(output);;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
              Popping language C
              Defined "LEVEL1_DCACHE_LINESIZE" to "64"
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <unistd.h>
long getconf_LEVEL1_DCACHE_ASSOC() { long val = sysconf(_SC_LEVEL1_DCACHE_ASSOC); return (0 <= val && val <= 2147483647) ? val : 2; }

int main() {
FILE *output = fopen("conftestval","w"); if (!output) return 1; fprintf(output,"%ld",getconf_LEVEL1_DCACHE_ASSOC()); fclose(output);;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
Executing: /tmp/petsc-VuReVk/config.utilities.cacheDetails/conftest
              Popping language C
              Defined "LEVEL1_DCACHE_ASSOC" to "8"
================================================================================
TEST check_siginfo_t from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:46)
TESTING: check_siginfo_t from config.types(config/BuildSystem/config/types.py:46)
  Checks if siginfo_t exists in signal.h. This check is for windows, and C89 check.
Checking for type: siginfo_t
                          All intermediate test results are stored in /tmp/petsc-VuReVk/config.types
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>
#include<signal.h>
#endif
    

int main() {
siginfo_t a;;
  return 0;
}
siginfo_t found
              Defined "HAVE_SIGINFO_T" to "1"
================================================================================
TEST check__int64 from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:52)
TESTING: check__int64 from config.types(config/BuildSystem/config/types.py:52)
  Checks if __int64 exists. This is primarily for windows.
Checking for type: __int64
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.types/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.types/conftest.c:13:1: error: unknown type name ???__int64???; did you mean ???__int64_t????
 __int64 a;;
 ^~~~~~~
 __int64_t
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
__int64 a;;
  return 0;
}
__int64 found
================================================================================
TEST checkSizeTypes from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:58)
TESTING: checkSizeTypes from config.types(config/BuildSystem/config/types.py:58)
  Checks for types associated with sizes, such as size_t.
Checking for type: size_t
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
size_t a;;
  return 0;
}
size_t found
================================================================================
TEST checkFileTypes from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:68)
TESTING: checkFileTypes from config.types(config/BuildSystem/config/types.py:68)
  Checks for types associated with files, such as mode_t, off_t, etc.
Checking for type: mode_t
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
mode_t a;;
  return 0;
}
mode_t found
Checking for type: off_t
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
off_t a;;
  return 0;
}
off_t found
================================================================================
TEST checkIntegerTypes from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:63)
TESTING: checkIntegerTypes from config.types(config/BuildSystem/config/types.py:63)
  Checks for types associated with integers, such as int32_t.
Checking for type: int32_t
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
int32_t a;;
  return 0;
}
int32_t found
================================================================================
TEST checkPID from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:74)
TESTING: checkPID from config.types(config/BuildSystem/config/types.py:74)
  Checks for pid_t, and defines it if necessary
Checking for type: pid_t
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stddef.h>

#endif
    

int main() {
pid_t a;;
  return 0;
}
pid_t found
================================================================================
TEST checkUID from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:78)
TESTING: checkUID from config.types(config/BuildSystem/config/types.py:78)
  Checks for uid_t and gid_t, and defines them if necessary
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/types.h>
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.types/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.types/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.types/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.types/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.types/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.types/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.types/conftest.c" 2
# 1 "/usr/include/sys/types.h" 1 3 4
# 25 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.types/conftest.c" 2
================================================================================
TEST checkSignal from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:85)
TESTING: checkSignal from config.types(config/BuildSystem/config/types.py:85)
  Checks the return type of signal() and defines RETSIGTYPE to that type name
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#include <signal.h>
#ifdef signal
#undef signal
#endif
#ifdef __cplusplus
extern "C" void (*signal (int, void(*)(int)))(int);
#else
void (*signal())();
#endif
    

int main() {
;
  return 0;
}
              Defined "RETSIGTYPE" to "void"
================================================================================
TEST checkC99Complex from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:106)
TESTING: checkC99Complex from config.types(config/BuildSystem/config/types.py:106)
  Check for complex numbers in in C99 std
       Note that since PETSc source code uses _Complex we test specifically for that, not complex
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <complex.h>

int main() {
double _Complex x;
 x = I;
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <complex.h>

int main() {
double _Complex x;
 x = I;
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
              Defined "HAVE_C99_COMPLEX" to "1"
================================================================================
TEST checkCxxComplex from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:117)
TESTING: checkCxxComplex from config.types(config/BuildSystem/config/types.py:117)
  Check for complex numbers in namespace std
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.types/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <complex>

int main() {
std::complex<double> x;
;
  return 0;
}
                      Pushing language CXX
                      Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
              Defined "HAVE_CXX_COMPLEX" to "1"
              Popping language Cxx
================================================================================
TEST checkFortranKind from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:138)
TESTING: checkFortranKind from config.types(config/BuildSystem/config/types.py:138)
  Checks whether selected_int_kind etc work USE_FORTRANKIND
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.types/conftest.F90 
Successful compile:
Source:
      program main

        integer(kind=selected_int_kind(10)) i
        real(kind=selected_real_kind(10)) d

      end
              Defined "USE_FORTRANKIND" to "1"
              Popping language FC
================================================================================
TEST checkConst from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:150)
TESTING: checkConst from config.types(config/BuildSystem/config/types.py:150)
  Checks for working const, and if not found defines it to empty string
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {

    /* Ultrix mips cc rejects this.  */
    typedef int charset[2]; const charset x;
    /* SunOS 4.1.1 cc rejects this.  */
    char const *const *ccp;
    char **p;
    /* NEC SVR4.0.2 mips cc rejects this.  */
    struct point {int x, y;};
    static struct point const zero = {0,0};
    /* AIX XL C 1.02.0.0 rejects this.
    It does not let you subtract one const X* pointer from another in an arm
    of an if-expression whose if-part is not a constant expression */
    const char *g = "string";
    ccp = &g + (g ? g-g : 0);
    /* HPUX 7.0 cc rejects these. */
    ++ccp;
    p = (char**) ccp;
    ccp = (char const *const *) p;
    /* This section avoids unused variable warnings */
    if (zero.x);
    if (x[0]);
    { /* SCO 3.2v4 cc rejects this.  */
      char *t;
      char const *s = 0 ? (char *) 0 : (char const *) 0;

      *t++ = 0;
      if (*s);
    }
    { /* Someone thinks the Sun supposedly-ANSI compiler will reject this.  */
      int x[] = {25, 17};
      const int *foo = &x[0];
      ++foo;
    }
    { /* Sun SC1.0 ANSI compiler rejects this -- but not the above. */
      typedef const int *iptr;
      iptr p = 0;
      ++p;
    }
    { /* AIX XL C 1.02.0.0 rejects this saying
      "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue. */
      struct s { int j; const int *ap[3]; };
      struct s *b; b->j = 5;
    }
    { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */
      const int foo = 10;

      /* Get rid of unused variable warning */
      if (foo);
    }
    ;
  return 0;
}
================================================================================
TEST checkEndian from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:206)
TESTING: checkEndian from config.types(config/BuildSystem/config/types.py:206)
  If the machine is big endian, defines WORDS_BIGENDIAN
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/types.h>
#ifdef HAVE_SYS_PARAM_H
  #include <sys/param.h>
#endif

int main() {

#if !BYTE_ORDER || !BIG_ENDIAN || !LITTLE_ENDIAN
  bogus endian macros
#endif
      ;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.types/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.types/conftest.c:11:3: error: unknown type name ???not???; did you mean ???ino_t????
   not big endian
   ^~~
   ino_t
/tmp/petsc-VuReVk/config.types/conftest.c:11:11: error: expected ???=???, ???,???, ???;???, ???asm??? or ???__attribute__??? before ???endian???
   not big endian
           ^~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/types.h>
#ifdef HAVE_SYS_PARAM_H
  #include <sys/param.h>
#endif

int main() {

#if BYTE_ORDER != BIG_ENDIAN
  not big endian
#endif
        ;
  return 0;
}
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: char
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(char));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_CHAR" to "1"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: void *
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(void *));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_VOID_P" to "8"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: short
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(short));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_SHORT" to "2"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: int
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(int));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_INT" to "4"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: long
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(long));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_LONG" to "8"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: long long
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(long long));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_LONG_LONG" to "8"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: float
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(float));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_FLOAT" to "4"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: double
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(double));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_DOUBLE" to "8"
================================================================================
TEST checkSizeof from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:259)
TESTING: checkSizeof from config.types(config/BuildSystem/config/types.py:259)
  Determines the size of type "typeName", and defines SIZEOF_"typeName" to be the size
Checking for size of type: size_t
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(size_t));
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                Popping language C
                Defined "SIZEOF_SIZE_T" to "8"
================================================================================
TEST checkBitsPerByte from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:310)
TESTING: checkBitsPerByte from config.types(config/BuildSystem/config/types.py:310)
  Determine the nubmer of bits per byte and define BITS_PER_BYTE
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#endif

int main() {
FILE *f = fopen("conftestval", "w");

    char val[2];
    int i = 0;

    if (!f) exit(1);
    val[0]='\1';
    val[1]='\0';
    while(val[0]) {val[0] <<= 1; i++;}
    fprintf(f, "%d\n", i);

    ;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
              Defined "BITS_PER_BYTE" to "8"
================================================================================
TEST checkVisibility from config.types(/home/ryan/projects/moose/petsc/config/BuildSystem/config/types.py:356)
TESTING: checkVisibility from config.types(config/BuildSystem/config/types.py:356)
              Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__attribute__((visibility ("default"))) int foo(void);;
  return 0;
}
              Defined "USE_VISIBILITY_C" to "1"
              Popping language C
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.types/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__attribute__((visibility ("default"))) int foo(void);;
  return 0;
}
              Defined "USE_VISIBILITY_CXX" to "1"
              Popping language Cxx
================================================================================
TEST configureMemAlign from PETSc.options.memAlign(/home/ryan/projects/moose/petsc/config/PETSc/options/memAlign.py:30)
TESTING: configureMemAlign from PETSc.options.memAlign(config/PETSc/options/memAlign.py:30)
  Choose alignment
              Defined "MEMALIGN" to "16"
            Memory alignment is 16
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
              Checking for functions [socket] in library ['socket', 'nsl'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char socket();
static void _check_socket() { socket(); }

int main() {
_check_socket();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lsocket -lnsl -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lsocket
collect2: error: ld returned 1 exit status
                Popping language C
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
              Checking for functions [handle_sigfpes] in library ['fpe'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char handle_sigfpes();
static void _check_handle_sigfpes() { handle_sigfpes(); }

int main() {
_check_handle_sigfpes();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lfpe -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfpe
collect2: error: ld returned 1 exit status
                Popping language C
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
              Checking for functions [socket] in library ['socket', 'nsl'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char socket();
static void _check_socket() { socket(); }

int main() {
_check_socket();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lsocket -lnsl -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lsocket
collect2: error: ld returned 1 exit status
                Popping language C
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
              Checking for functions [handle_sigfpes] in library ['fpe'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char handle_sigfpes();
static void _check_handle_sigfpes() { handle_sigfpes(); }

int main() {
_check_handle_sigfpes();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lfpe -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfpe
collect2: error: ld returned 1 exit status
                Popping language C
================================================================================
TEST checkMath from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:265)
TESTING: checkMath from config.libraries(config/BuildSystem/config/libraries.py:265)
  Check for sin() in libm, the math library
              Checking for functions [sin floor log10 pow] in library [''] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <stdio.h>
double sin(double);
static void _check_sin() { double x,y; scanf("%lf",&x); y = sin(x); printf("%f",y);
; }
#include <stdio.h>
double floor(double);
static void _check_floor() { double x,y; scanf("%lf",&x); y = floor(x); printf("%f",y);
; }
#include <stdio.h>
double log10(double);
static void _check_log10() { double x,y; scanf("%lf",&x); y = log10(x); printf("%f",y);
; }
#include <stdio.h>
double pow(double, double);
static void _check_pow() { double x,y; scanf("%lf",&x); y = pow(x,x); printf("%f",y);
; }

int main() {
_check_sin();
_check_floor();
_check_log10();
_check_pow();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o   -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: undefined reference to symbol 'log10@@GLIBC_2.2.5'
/usr/bin/ld: /usr/lib/libm.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
                Popping language C
              Checking for functions [sin floor log10 pow] in library ['m'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <stdio.h>
double sin(double);
static void _check_sin() { double x,y; scanf("%lf",&x); y = sin(x); printf("%f",y);
; }
#include <stdio.h>
double floor(double);
static void _check_floor() { double x,y; scanf("%lf",&x); y = floor(x); printf("%f",y);
; }
#include <stdio.h>
double log10(double);
static void _check_log10() { double x,y; scanf("%lf",&x); y = log10(x); printf("%f",y);
; }
#include <stdio.h>
double pow(double, double);
static void _check_pow() { double x,y; scanf("%lf",&x); y = pow(x,x); printf("%f",y);
; }

int main() {
_check_sin();
_check_floor();
_check_log10();
_check_pow();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lm -lstdc++ -ldl 
                Defined "HAVE_LIBM" to "1"
                Popping language C
            CheckMath: using math library ['libm.a']
================================================================================
TEST checkMathErf from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:284)
TESTING: checkMathErf from config.libraries(config/BuildSystem/config/libraries.py:284)
  Check for erf() in libm, the math library
              Checking for functions [erf] in library ['libm.a'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <math.h>
static void _check_erf() { double (*checkErf)(double) = erf;double x = 0,y; y = (*checkErf)(x); }

int main() {
_check_erf();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lm -lstdc++ -ldl 
                Defined "HAVE_LIBM" to "1"
                Popping language C
            erf() found
              Defined "HAVE_ERF" to "1"
================================================================================
TEST checkMathTgamma from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:293)
TESTING: checkMathTgamma from config.libraries(config/BuildSystem/config/libraries.py:293)
  Check for tgamma() in libm, the math library
              Checking for functions [tgamma] in library ['libm.a'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <math.h>
static void _check_tgamma() { double (*checkTgamma)(double) = tgamma;double x = 0,y; y = (*checkTgamma)(x); }

int main() {
_check_tgamma();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lm -lstdc++ -ldl 
                Defined "HAVE_LIBM" to "1"
                Popping language C
            tgamma() found
              Defined "HAVE_TGAMMA" to "1"
================================================================================
TEST checkMathFenv from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:302)
TESTING: checkMathFenv from config.libraries(config/BuildSystem/config/libraries.py:302)
  Checks if <fenv.h> can be used with FE_DFL_ENV
              Checking for functions [fesetenv] in library ['libm.a'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <fenv.h>
static void _check_fesetenv() { fesetenv(FE_DFL_ENV);; }

int main() {
_check_fesetenv();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lm -lstdc++ -ldl 
                Defined "HAVE_LIBM" to "1"
                Popping language C
              Defined "HAVE_FENV_H" to "1"
================================================================================
TEST checkMathLog2 from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:310)
TESTING: checkMathLog2 from config.libraries(config/BuildSystem/config/libraries.py:310)
  Check for log2() in libm, the math library
              Checking for functions [log2] in library ['libm.a'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <math.h>
static void _check_log2() { double (*checkLog2)(double) = log2; double x = 2.5, y = (*checkLog2)(x); }

int main() {
_check_log2();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lm -lstdc++ -ldl 
                Defined "HAVE_LIBM" to "1"
                Popping language C
            log2() found
              Defined "HAVE_LOG2" to "1"
================================================================================
TEST checkRealtime from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:319)
TESTING: checkRealtime from config.libraries(config/BuildSystem/config/libraries.py:319)
  Check for presence of clock_gettime() in realtime library (POSIX Realtime extensions)
              Checking for functions [clock_gettime] in library [''] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <time.h>
static void _check_clock_gettime() { struct timespec tp; clock_gettime(CLOCK_REALTIME,&tp);; }

int main() {
_check_clock_gettime();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o   -lstdc++ -ldl 
                Popping language C
            realtime functions are linked in by default
================================================================================
TEST checkDynamic from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:335)
TESTING: checkDynamic from config.libraries(config/BuildSystem/config/libraries.py:335)
  Check for the header and libraries necessary for dynamic library manipulation
              Checking for functions [dlopen] in library ['dl'] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char dlopen();
static void _check_dlopen() { dlopen(); }

int main() {
_check_dlopen();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -ldl -lstdc++ -ldl 
                Defined "HAVE_LIBDL" to "1"
                Popping language C
Checking for header: dlfcn.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/dlfcn.h" 1 3 4
# 22 "/usr/include/dlfcn.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 23 "/usr/include/dlfcn.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 25 "/usr/include/dlfcn.h" 2 3 4
# 1 "/usr/include/bits/dlfcn.h" 1 3 4
# 28 "/usr/include/dlfcn.h" 2 3 4
# 52 "/usr/include/dlfcn.h" 3 4
extern void *dlopen (const char *__file, int __mode) __attribute__ ((__nothrow__));
extern int dlclose (void *__handle) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern void *dlsym (void *__restrict __handle,
      const char *__restrict __name) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 82 "/usr/include/dlfcn.h" 3 4
extern char *dlerror (void) __attribute__ ((__nothrow__ , __leaf__));
# 188 "/usr/include/dlfcn.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_DLFCN_H" to "1"
================================================================================
TEST configureLibraryOptions from PETSc.options.libraryOptions(/home/ryan/projects/moose/petsc/config/PETSc/options/libraryOptions.py:38)
TESTING: configureLibraryOptions from PETSc.options.libraryOptions(config/PETSc/options/libraryOptions.py:38)
  Sets PETSC_USE_DEBUG, PETSC_USE_INFO, PETSC_USE_LOG, PETSC_USE_CTABLE, PETSC_USE_FORTRAN_KERNELS, and PETSC_USE_AVX512_KERNELS
              Defined "USE_LOG" to "1"
Executing: mpicc -qversion
              Defined "USE_MALLOC_COALESCED" to "1"
              Defined "USE_INFO" to "1"
              Defined "USE_CTABLE" to "1"
              Defined "USE_BACKWARD_LOOP" to "1"
              **********Checking if running on BGL/IBM detected
                Checking for functions [bgl_perfctr_void] in library [''] []
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char bgl_perfctr_void();
static void _check_bgl_perfctr_void() { bgl_perfctr_void(); }

int main() {
_check_bgl_perfctr_void();;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o   -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_bgl_perfctr_void':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `bgl_perfctr_void'
collect2: error: ld returned 1 exit status
                  Popping language C
                Checking for functions [ADIOI_BGL_Open] in library [''] []
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ADIOI_BGL_Open();
static void _check_ADIOI_BGL_Open() { ADIOI_BGL_Open(); }

int main() {
_check_ADIOI_BGL_Open();;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o   -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_ADIOI_BGL_Open':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `ADIOI_BGL_Open'
collect2: error: ld returned 1 exit status
                  Popping language C
              *********BGL/IBM test failure
              Defined "Alignx(a,b)" to "  "
================================================================================
TEST configureISColorValueType from PETSc.options.libraryOptions(/home/ryan/projects/moose/petsc/config/PETSc/options/libraryOptions.py:96)
TESTING: configureISColorValueType from PETSc.options.libraryOptions(config/PETSc/options/libraryOptions.py:96)
  Sets PETSC_IS_COLOR_VALUE_TYPE, MPIU_COLORING_VALUE, IS_COLORING_MAX required by ISColor
              Defined "MPIU_COLORING_VALUE" to "MPI_UNSIGNED_SHORT"
              Defined "IS_COLORING_MAX" to "65535"
              Defined "IS_COLOR_VALUE_TYPE" to "short"
              Defined "IS_COLOR_VALUE_TYPE_F" to "integer2"
================================================================================
TEST configureCPURelax from config.atomics(/home/ryan/projects/moose/petsc/config/BuildSystem/config/atomics.py:17)
TESTING: configureCPURelax from config.atomics(config/BuildSystem/config/atomics.py:17)
   Definitions for cpu relax assembly instructions 
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.atomics
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__("rep; nop" ::: "memory");;
  return 0;
}
              Defined "CPU_RELAX()" to "__asm__ __volatile__("rep; nop" ::: "memory")"
================================================================================
TEST configureMemoryBarriers from config.atomics(/home/ryan/projects/moose/petsc/config/BuildSystem/config/atomics.py:38)
TESTING: configureMemoryBarriers from config.atomics(config/BuildSystem/config/atomics.py:38)
   Definitions for memory barrier instructions
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("mfence":::"memory");
  return 0;
}
              Defined "MEMORY_BARRIER()" to "__asm__ __volatile__ ("mfence":::"memory")"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("lfence":::"memory");
  return 0;
}
              Defined "READ_MEMORY_BARRIER()" to "__asm__ __volatile__ ("lfence":::"memory")"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("sfence":::"memory");
  return 0;
}
              Defined "WRITE_MEMORY_BARRIER()" to "__asm__ __volatile__ ("sfence":::"memory")"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `sync'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("sync":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `lwsync'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("lwsync":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `eieio'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("eieio":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `dmb'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("dmb":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `dmb ish'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("dmb ish":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `dmb ishld'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("dmb ishld":::"memory");
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.atomics/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.atomics/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.atomics/conftest.c: Assembler messages:
/tmp/petsc-VuReVk/config.atomics/conftest.c:5: Error: no such instruction: `dmb ishst'
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
__asm__ __volatile__ ("dmb ishst":::"memory");
  return 0;
}
================================================================================
TEST checkMemcmp from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:110)
TESTING: checkMemcmp from config.functions(config/BuildSystem/config/functions.py:110)
  Check for 8-bit clean memcmp
            Making executable to test memcmp()
                              All intermediate test results are stored in /tmp/petsc-VuReVk/config.functions
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <string.h>
void exit(int);


int main() {
char c0 = 0x40;
char c1 = (char) 0x80;
char c2 = (char) 0x81;
exit(memcmp(&c0, &c2, 1) < 0 && memcmp(&c1, &c2, 1) < 0 ? 0 : 1);
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.functions/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.functions/conftest
Executing: /tmp/petsc-VuReVk/config.functions/conftest
================================================================================
TEST checkSysinfo from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:135)
TESTING: checkSysinfo from config.functions(config/BuildSystem/config/functions.py:135)
  Check whether sysinfo takes three arguments, and if it does define HAVE_SYSINFO_3ARG
Checking for functions [sysinfo]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char sysinfo();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_sysinfo) || defined (__stub___sysinfo)
sysinfo_will_always_fail_with_ENOSYS();
#else
sysinfo();
#endif
;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                Defined "HAVE_SYSINFO" to "1"
Checking for header: linux/kernel.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/linux/kernel.h" 1 3 4
# 1 "/usr/include/linux/sysinfo.h" 1 3 4
# 1 "/usr/include/linux/types.h" 1 3 4
# 1 "/usr/include/asm/types.h" 1 3 4
# 1 "/usr/include/asm-generic/types.h" 1 3 4
# 1 "/usr/include/asm-generic/int-ll64.h" 1 3 4
# 12 "/usr/include/asm-generic/int-ll64.h" 3 4
# 1 "/usr/include/asm/bitsperlong.h" 1 3 4
# 11 "/usr/include/asm/bitsperlong.h" 3 4
# 1 "/usr/include/asm-generic/bitsperlong.h" 1 3 4
# 12 "/usr/include/asm/bitsperlong.h" 2 3 4
# 13 "/usr/include/asm-generic/int-ll64.h" 2 3 4
# 20 "/usr/include/asm-generic/int-ll64.h" 3 4
typedef __signed__ char __s8;
typedef unsigned char __u8;
typedef __signed__ short __s16;
typedef unsigned short __u16;
typedef __signed__ int __s32;
typedef unsigned int __u32;
__extension__ typedef __signed__ long long __s64;
__extension__ typedef unsigned long long __u64;
# 8 "/usr/include/asm-generic/types.h" 2 3 4
# 6 "/usr/include/asm/types.h" 2 3 4
# 6 "/usr/include/linux/types.h" 2 3 4
# 1 "/usr/include/linux/posix_types.h" 1 3 4
# 1 "/usr/include/linux/stddef.h" 1 3 4
# 6 "/usr/include/linux/posix_types.h" 2 3 4
# 25 "/usr/include/linux/posix_types.h" 3 4
typedef struct {
 unsigned long fds_bits[1024 / (8 * sizeof(long))];
} __kernel_fd_set;
typedef void (*__kernel_sighandler_t)(int);
typedef int __kernel_key_t;
typedef int __kernel_mqd_t;
# 1 "/usr/include/asm/posix_types.h" 1 3 4
# 1 "/usr/include/asm/posix_types_64.h" 1 3 4
# 11 "/usr/include/asm/posix_types_64.h" 3 4
typedef unsigned short __kernel_old_uid_t;
typedef unsigned short __kernel_old_gid_t;
typedef unsigned long __kernel_old_dev_t;
# 1 "/usr/include/asm-generic/posix_types.h" 1 3 4
# 15 "/usr/include/asm-generic/posix_types.h" 3 4
typedef long __kernel_long_t;
typedef unsigned long __kernel_ulong_t;
typedef __kernel_ulong_t __kernel_ino_t;
typedef unsigned int __kernel_mode_t;
typedef int __kernel_pid_t;
typedef int __kernel_ipc_pid_t;
typedef unsigned int __kernel_uid_t;
typedef unsigned int __kernel_gid_t;
typedef __kernel_long_t __kernel_suseconds_t;
typedef int __kernel_daddr_t;
typedef unsigned int __kernel_uid32_t;
typedef unsigned int __kernel_gid32_t;
# 72 "/usr/include/asm-generic/posix_types.h" 3 4
typedef __kernel_ulong_t __kernel_size_t;
typedef __kernel_long_t __kernel_ssize_t;
typedef __kernel_long_t __kernel_ptrdiff_t;
typedef struct {
 int val[2];
} __kernel_fsid_t;
typedef __kernel_long_t __kernel_off_t;
typedef long long __kernel_loff_t;
typedef __kernel_long_t __kernel_time_t;
typedef __kernel_long_t __kernel_clock_t;
typedef int __kernel_timer_t;
typedef int __kernel_clockid_t;
typedef char * __kernel_caddr_t;
typedef unsigned short __kernel_uid16_t;
typedef unsigned short __kernel_gid16_t;
# 19 "/usr/include/asm/posix_types_64.h" 2 3 4
# 8 "/usr/include/asm/posix_types.h" 2 3 4
# 37 "/usr/include/linux/posix_types.h" 2 3 4
# 10 "/usr/include/linux/types.h" 2 3 4
# 24 "/usr/include/linux/types.h" 3 4
typedef __u16 __le16;
typedef __u16 __be16;
typedef __u32 __le32;
typedef __u32 __be32;
typedef __u64 __le64;
typedef __u64 __be64;
typedef __u16 __sum16;
typedef __u32 __wsum;
# 50 "/usr/include/linux/types.h" 3 4
typedef unsigned __poll_t;
# 6 "/usr/include/linux/sysinfo.h" 2 3 4
struct sysinfo {
 __kernel_long_t uptime;
 __kernel_ulong_t loads[3];
 __kernel_ulong_t totalram;
 __kernel_ulong_t freeram;
 __kernel_ulong_t sharedram;
 __kernel_ulong_t bufferram;
 __kernel_ulong_t totalswap;
 __kernel_ulong_t freeswap;
 __u16 procs;
 __u16 pad;
 __kernel_ulong_t totalhigh;
 __kernel_ulong_t freehigh;
 __u32 mem_unit;
 char _f[20-2*sizeof(__kernel_ulong_t)-sizeof(__u32)];
};
# 6 "/usr/include/linux/kernel.h" 2 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_LINUX_KERNEL_H" to "1"
Checking for header: sys/sysinfo.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/sys/sysinfo.h" 1 3 4
# 21 "/usr/include/sys/sysinfo.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/sys/sysinfo.h" 2 3 4
# 1 "/usr/include/linux/kernel.h" 1 3 4
# 1 "/usr/include/linux/sysinfo.h" 1 3 4
# 1 "/usr/include/linux/types.h" 1 3 4
# 1 "/usr/include/asm/types.h" 1 3 4
# 1 "/usr/include/asm-generic/types.h" 1 3 4
# 1 "/usr/include/asm-generic/int-ll64.h" 1 3 4
# 12 "/usr/include/asm-generic/int-ll64.h" 3 4
# 1 "/usr/include/asm/bitsperlong.h" 1 3 4
# 11 "/usr/include/asm/bitsperlong.h" 3 4
# 1 "/usr/include/asm-generic/bitsperlong.h" 1 3 4
# 12 "/usr/include/asm/bitsperlong.h" 2 3 4
# 13 "/usr/include/asm-generic/int-ll64.h" 2 3 4
# 20 "/usr/include/asm-generic/int-ll64.h" 3 4
typedef __signed__ char __s8;
typedef unsigned char __u8;
typedef __signed__ short __s16;
typedef unsigned short __u16;
typedef __signed__ int __s32;
typedef unsigned int __u32;
__extension__ typedef __signed__ long long __s64;
__extension__ typedef unsigned long long __u64;
# 8 "/usr/include/asm-generic/types.h" 2 3 4
# 6 "/usr/include/asm/types.h" 2 3 4
# 6 "/usr/include/linux/types.h" 2 3 4
# 1 "/usr/include/linux/posix_types.h" 1 3 4
# 1 "/usr/include/linux/stddef.h" 1 3 4
# 6 "/usr/include/linux/posix_types.h" 2 3 4
# 25 "/usr/include/linux/posix_types.h" 3 4
typedef struct {
 unsigned long fds_bits[1024 / (8 * sizeof(long))];
} __kernel_fd_set;
typedef void (*__kernel_sighandler_t)(int);
typedef int __kernel_key_t;
typedef int __kernel_mqd_t;
# 1 "/usr/include/asm/posix_types.h" 1 3 4
# 1 "/usr/include/asm/posix_types_64.h" 1 3 4
# 11 "/usr/include/asm/posix_types_64.h" 3 4
typedef unsigned short __kernel_old_uid_t;
typedef unsigned short __kernel_old_gid_t;
typedef unsigned long __kernel_old_dev_t;
# 1 "/usr/include/asm-generic/posix_types.h" 1 3 4
# 15 "/usr/include/asm-generic/posix_types.h" 3 4
typedef long __kernel_long_t;
typedef unsigned long __kernel_ulong_t;
typedef __kernel_ulong_t __kernel_ino_t;
typedef unsigned int __kernel_mode_t;
typedef int __kernel_pid_t;
typedef int __kernel_ipc_pid_t;
typedef unsigned int __kernel_uid_t;
typedef unsigned int __kernel_gid_t;
typedef __kernel_long_t __kernel_suseconds_t;
typedef int __kernel_daddr_t;
typedef unsigned int __kernel_uid32_t;
typedef unsigned int __kernel_gid32_t;
# 72 "/usr/include/asm-generic/posix_types.h" 3 4
typedef __kernel_ulong_t __kernel_size_t;
typedef __kernel_long_t __kernel_ssize_t;
typedef __kernel_long_t __kernel_ptrdiff_t;
typedef struct {
 int val[2];
} __kernel_fsid_t;
typedef __kernel_long_t __kernel_off_t;
typedef long long __kernel_loff_t;
typedef __kernel_long_t __kernel_time_t;
typedef __kernel_long_t __kernel_clock_t;
typedef int __kernel_timer_t;
typedef int __kernel_clockid_t;
typedef char * __kernel_caddr_t;
typedef unsigned short __kernel_uid16_t;
typedef unsigned short __kernel_gid16_t;
# 19 "/usr/include/asm/posix_types_64.h" 2 3 4
# 8 "/usr/include/asm/posix_types.h" 2 3 4
# 37 "/usr/include/linux/posix_types.h" 2 3 4
# 10 "/usr/include/linux/types.h" 2 3 4
# 24 "/usr/include/linux/types.h" 3 4
typedef __u16 __le16;
typedef __u16 __be16;
typedef __u32 __le32;
typedef __u32 __be32;
typedef __u64 __le64;
typedef __u64 __be64;
typedef __u16 __sum16;
typedef __u32 __wsum;
# 50 "/usr/include/linux/types.h" 3 4
typedef unsigned __poll_t;
# 6 "/usr/include/linux/sysinfo.h" 2 3 4
struct sysinfo {
 __kernel_long_t uptime;
 __kernel_ulong_t loads[3];
 __kernel_ulong_t totalram;
 __kernel_ulong_t freeram;
 __kernel_ulong_t sharedram;
 __kernel_ulong_t bufferram;
 __kernel_ulong_t totalswap;
 __kernel_ulong_t freeswap;
 __u16 procs;
 __u16 pad;
 __kernel_ulong_t totalhigh;
 __kernel_ulong_t freehigh;
 __u32 mem_unit;
 char _f[20-2*sizeof(__kernel_ulong_t)-sizeof(__u32)];
};
# 6 "/usr/include/linux/kernel.h" 2 3 4
# 25 "/usr/include/sys/sysinfo.h" 2 3 4
extern int sysinfo (struct sysinfo *__info) __attribute__ ((__nothrow__ , __leaf__));
extern int get_nprocs_conf (void) __attribute__ ((__nothrow__ , __leaf__));
extern int get_nprocs (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int get_phys_pages (void) __attribute__ ((__nothrow__ , __leaf__));
extern long int get_avphys_pages (void) __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
                Defined "HAVE_SYS_SYSINFO_H" to "1"
Checking for header: sys/systeminfo.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory
 #include <sys/systeminfo.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/systeminfo.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory
 #include <sys/systeminfo.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sys/systeminfo.h: No such file or directory #include <sys/systeminfo.h>          ^~~~~~~~~~~~~~~~~~compilation terminated.:
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:13:4: error: #error "Cannot check sysinfo without special headers"
 #  error "Cannot check sysinfo without special headers"
    ^~~~~
/tmp/petsc-VuReVk/config.functions/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.functions/conftest.c:17:30: warning: implicit declaration of function ???sysinfo??? [-Wimplicit-function-declaration]
 char buf[10]; long count=10; sysinfo(1, buf, count);
                              ^~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

#ifdef HAVE_LINUX_KERNEL_H
#  include <linux/kernel.h>
#  include <linux/sys.h>
#  ifdef HAVE_SYS_SYSINFO_H
#    include <sys/sysinfo.h>
#  endif
#elif defined(HAVE_SYS_SYSTEMINFO_H)
#  include <sys/systeminfo.h>
#else
#  error "Cannot check sysinfo without special headers"
#endif

int main() {
char buf[10]; long count=10; sysinfo(1, buf, count);
;
  return 0;
}
                Compile failed inside link

================================================================================
TEST checkVPrintf from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:158)
TESTING: checkVPrintf from config.functions(config/BuildSystem/config/functions.py:158)
  Checks whether vprintf requires a char * last argument, and if it does defines HAVE_VPRINTF_CHAR
Checking for functions [vprintf]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:13:6: warning: conflicting types for built-in function ???vprintf??? [-Wbuiltin-declaration-mismatch]
 char vprintf();
      ^~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char vprintf();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_vprintf) || defined (__stub___vprintf)
vprintf_will_always_fail_with_ENOSYS();
#else
vprintf();
#endif
;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                Defined "HAVE_VPRINTF" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <stdarg.h>

int main() {
va_list Argp;
vprintf( "%d", Argp );
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
================================================================================
TEST checkVFPrintf from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:165)
TESTING: checkVFPrintf from config.functions(config/BuildSystem/config/functions.py:165)
  Checks whether vfprintf requires a char * last argument, and if it does defines HAVE_VFPRINTF_CHAR
Checking for functions [vfprintf]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:13:6: warning: conflicting types for built-in function ???vfprintf??? [-Wbuiltin-declaration-mismatch]
 char vfprintf();
      ^~~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char vfprintf();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_vfprintf) || defined (__stub___vfprintf)
vfprintf_will_always_fail_with_ENOSYS();
#else
vfprintf();
#endif
;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                Defined "HAVE_VFPRINTF" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <stdarg.h>

int main() {
va_list Argp;
vfprintf(stdout, "%d", Argp );
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
================================================================================
TEST checkVSNPrintf from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:172)
TESTING: checkVSNPrintf from config.functions(config/BuildSystem/config/functions.py:172)
  Checks whether vsnprintf requires a char * last argument, and if it does defines HAVE_VSNPRINTF_CHAR
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdio.h>
#include <stdarg.h>

int main() {
va_list Argp;char str[6];
vsnprintf(str,5, "%d", Argp );
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "HAVE_VSNPRINTF" to "1"
================================================================================
TEST checkNanosleep from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:202)
TESTING: checkNanosleep from config.functions(config/BuildSystem/config/functions.py:202)
  Check for functional nanosleep() - as time.h behaves differently for different compiler flags - like -std=c89
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <time.h>

int main() {
struct timespec tp;
 tp.tv_sec = 0;
 tp.tv_nsec = (long)(1e9);
 nanosleep(&tp,0);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "HAVE_NANOSLEEP" to "1"
================================================================================
TEST checkMemmove from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:208)
TESTING: checkMemmove from config.functions(config/BuildSystem/config/functions.py:208)
  Check for functional memmove() - as MS VC requires correct includes to for this test
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <string.h>

int main() {
 char c1[1], c2[1] = "c";
 size_t n=1;
 memmove(c1,c2,n);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "HAVE_MEMMOVE" to "1"
================================================================================
TEST checkSignalHandlerType from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:178)
TESTING: checkSignalHandlerType from config.functions(config/BuildSystem/config/functions.py:178)
  Checks the type of C++ signals handlers, and defines SIGNAL_CAST to the correct value
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.functions/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>
static void myhandler(int sig) {}

int main() {
signal(SIGFPE,myhandler);
;
  return 0;
}
                      Pushing language CXX
                      Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "SIGNAL_CAST" to " "
              Popping language Cxx
================================================================================
TEST checkFreeReturnType from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:188)
TESTING: checkFreeReturnType from config.functions(config/BuildSystem/config/functions.py:188)
  Checks whether free returns void or int, and defines HAVE_FREE_RETURN_INT
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.functions/conftest.c:6:25: error: void value not ignored as it ought to be
 int ierr; void *p; ierr = free(p); return 0;
                         ^
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdlib.h>

int main() {
int ierr; void *p; ierr = free(p); return 0;
;
  return 0;
}
                Compile failed inside link

================================================================================
TEST checkVariableArgumentLists from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:194)
TESTING: checkVariableArgumentLists from config.functions(config/BuildSystem/config/functions.py:194)
  Checks whether the variable argument list functionality is working
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <stdarg.h>

int main() {
  va_list l1, l2;
  va_copy(l1, l2);
  return 0;
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "HAVE_VA_COPY" to "1"
================================================================================
TEST checkClassify from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:89)
TESTING: checkClassify from config.functions(config/BuildSystem/config/functions.py:89)
  Recursive decompose to rapidly classify functions as found or missing
    To confirm that a function is missing, we require a compile/link
    failure with only that function in a compilation unit.  In contrast,
    we can confirm that many functions are present by compiling them all
    together in a large compilation unit.  We optimistically compile
    everything together, then trim all functions that were named in the
    error message and bisect the result.  The trimming is only an
    optimization to increase the likelihood of a big-batch compile
    succeeding; we do not rely on the compiler naming missing functions.
    
Checking for functions [rand getdomainname _sleep snprintf realpath dlsym bzero _getcwd getwd uname _lseek sleep _access lseek usleep dlclose gethostname clock get_nprocs access _snprintf dlerror mkstemp fork getpagesize sbreak memalign sigset getcwd gethostbyname gettimeofday readlink _set_output_format PXFGETARG sigaction strcasecmp dlopen drand48 socket signal popen getrusage times _mkdir time sysctlbyname stricmp]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:16:6: warning: conflicting types for built-in function ???snprintf??? [-Wbuiltin-declaration-mismatch]
 char snprintf();
      ^~~~~~~~
/tmp/petsc-VuReVk/config.functions/conftest.c:19:6: warning: conflicting types for built-in function ???bzero??? [-Wbuiltin-declaration-mismatch]
 char bzero();
      ^~~~~
/tmp/petsc-VuReVk/config.functions/conftest.c:36:6: warning: conflicting types for built-in function ???fork??? [-Wbuiltin-declaration-mismatch]
 char fork();
      ^~~~
/tmp/petsc-VuReVk/config.functions/conftest.c:48:6: warning: conflicting types for built-in function ???strcasecmp??? [-Wbuiltin-declaration-mismatch]
 char strcasecmp();
      ^~~~~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char rand();
char getdomainname();
char _sleep();
char snprintf();
char realpath();
char dlsym();
char bzero();
char _getcwd();
char getwd();
char uname();
char _lseek();
char sleep();
char _access();
char lseek();
char usleep();
char dlclose();
char gethostname();
char clock();
char get_nprocs();
char access();
char _snprintf();
char dlerror();
char mkstemp();
char fork();
char getpagesize();
char sbreak();
char memalign();
char sigset();
char getcwd();
char gethostbyname();
char gettimeofday();
char readlink();
char _set_output_format();
char PXFGETARG();
char sigaction();
char strcasecmp();
char dlopen();
char drand48();
char socket();
char signal();
char popen();
char getrusage();
char times();
char _mkdir();
char time();
char sysctlbyname();
char stricmp();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_rand) || defined (__stub___rand)
rand_will_always_fail_with_ENOSYS();
#else
rand();
#endif

#if defined (__stub_getdomainname) || defined (__stub___getdomainname)
getdomainname_will_always_fail_with_ENOSYS();
#else
getdomainname();
#endif

#if defined (__stub__sleep) || defined (__stub____sleep)
_sleep_will_always_fail_with_ENOSYS();
#else
_sleep();
#endif

#if defined (__stub_snprintf) || defined (__stub___snprintf)
snprintf_will_always_fail_with_ENOSYS();
#else
snprintf();
#endif

#if defined (__stub_realpath) || defined (__stub___realpath)
realpath_will_always_fail_with_ENOSYS();
#else
realpath();
#endif

#if defined (__stub_dlsym) || defined (__stub___dlsym)
dlsym_will_always_fail_with_ENOSYS();
#else
dlsym();
#endif

#if defined (__stub_bzero) || defined (__stub___bzero)
bzero_will_always_fail_with_ENOSYS();
#else
bzero();
#endif

#if defined (__stub__getcwd) || defined (__stub____getcwd)
_getcwd_will_always_fail_with_ENOSYS();
#else
_getcwd();
#endif

#if defined (__stub_getwd) || defined (__stub___getwd)
getwd_will_always_fail_with_ENOSYS();
#else
getwd();
#endif

#if defined (__stub_uname) || defined (__stub___uname)
uname_will_always_fail_with_ENOSYS();
#else
uname();
#endif

#if defined (__stub__lseek) || defined (__stub____lseek)
_lseek_will_always_fail_with_ENOSYS();
#else
_lseek();
#endif

#if defined (__stub_sleep) || defined (__stub___sleep)
sleep_will_always_fail_with_ENOSYS();
#else
sleep();
#endif

#if defined (__stub__access) || defined (__stub____access)
_access_will_always_fail_with_ENOSYS();
#else
_access();
#endif

#if defined (__stub_lseek) || defined (__stub___lseek)
lseek_will_always_fail_with_ENOSYS();
#else
lseek();
#endif

#if defined (__stub_usleep) || defined (__stub___usleep)
usleep_will_always_fail_with_ENOSYS();
#else
usleep();
#endif

#if defined (__stub_dlclose) || defined (__stub___dlclose)
dlclose_will_always_fail_with_ENOSYS();
#else
dlclose();
#endif

#if defined (__stub_gethostname) || defined (__stub___gethostname)
gethostname_will_always_fail_with_ENOSYS();
#else
gethostname();
#endif

#if defined (__stub_clock) || defined (__stub___clock)
clock_will_always_fail_with_ENOSYS();
#else
clock();
#endif

#if defined (__stub_get_nprocs) || defined (__stub___get_nprocs)
get_nprocs_will_always_fail_with_ENOSYS();
#else
get_nprocs();
#endif

#if defined (__stub_access) || defined (__stub___access)
access_will_always_fail_with_ENOSYS();
#else
access();
#endif

#if defined (__stub__snprintf) || defined (__stub____snprintf)
_snprintf_will_always_fail_with_ENOSYS();
#else
_snprintf();
#endif

#if defined (__stub_dlerror) || defined (__stub___dlerror)
dlerror_will_always_fail_with_ENOSYS();
#else
dlerror();
#endif

#if defined (__stub_mkstemp) || defined (__stub___mkstemp)
mkstemp_will_always_fail_with_ENOSYS();
#else
mkstemp();
#endif

#if defined (__stub_fork) || defined (__stub___fork)
fork_will_always_fail_with_ENOSYS();
#else
fork();
#endif

#if defined (__stub_getpagesize) || defined (__stub___getpagesize)
getpagesize_will_always_fail_with_ENOSYS();
#else
getpagesize();
#endif

#if defined (__stub_sbreak) || defined (__stub___sbreak)
sbreak_will_always_fail_with_ENOSYS();
#else
sbreak();
#endif

#if defined (__stub_memalign) || defined (__stub___memalign)
memalign_will_always_fail_with_ENOSYS();
#else
memalign();
#endif

#if defined (__stub_sigset) || defined (__stub___sigset)
sigset_will_always_fail_with_ENOSYS();
#else
sigset();
#endif

#if defined (__stub_getcwd) || defined (__stub___getcwd)
getcwd_will_always_fail_with_ENOSYS();
#else
getcwd();
#endif

#if defined (__stub_gethostbyname) || defined (__stub___gethostbyname)
gethostbyname_will_always_fail_with_ENOSYS();
#else
gethostbyname();
#endif

#if defined (__stub_gettimeofday) || defined (__stub___gettimeofday)
gettimeofday_will_always_fail_with_ENOSYS();
#else
gettimeofday();
#endif

#if defined (__stub_readlink) || defined (__stub___readlink)
readlink_will_always_fail_with_ENOSYS();
#else
readlink();
#endif

#if defined (__stub__set_output_format) || defined (__stub____set_output_format)
_set_output_format_will_always_fail_with_ENOSYS();
#else
_set_output_format();
#endif

#if defined (__stub_PXFGETARG) || defined (__stub___PXFGETARG)
PXFGETARG_will_always_fail_with_ENOSYS();
#else
PXFGETARG();
#endif

#if defined (__stub_sigaction) || defined (__stub___sigaction)
sigaction_will_always_fail_with_ENOSYS();
#else
sigaction();
#endif

#if defined (__stub_strcasecmp) || defined (__stub___strcasecmp)
strcasecmp_will_always_fail_with_ENOSYS();
#else
strcasecmp();
#endif

#if defined (__stub_dlopen) || defined (__stub___dlopen)
dlopen_will_always_fail_with_ENOSYS();
#else
dlopen();
#endif

#if defined (__stub_drand48) || defined (__stub___drand48)
drand48_will_always_fail_with_ENOSYS();
#else
drand48();
#endif

#if defined (__stub_socket) || defined (__stub___socket)
socket_will_always_fail_with_ENOSYS();
#else
socket();
#endif

#if defined (__stub_signal) || defined (__stub___signal)
signal_will_always_fail_with_ENOSYS();
#else
signal();
#endif

#if defined (__stub_popen) || defined (__stub___popen)
popen_will_always_fail_with_ENOSYS();
#else
popen();
#endif

#if defined (__stub_getrusage) || defined (__stub___getrusage)
getrusage_will_always_fail_with_ENOSYS();
#else
getrusage();
#endif

#if defined (__stub_times) || defined (__stub___times)
times_will_always_fail_with_ENOSYS();
#else
times();
#endif

#if defined (__stub__mkdir) || defined (__stub____mkdir)
_mkdir_will_always_fail_with_ENOSYS();
#else
_mkdir();
#endif

#if defined (__stub_time) || defined (__stub___time)
time_will_always_fail_with_ENOSYS();
#else
time();
#endif

#if defined (__stub_sysctlbyname) || defined (__stub___sysctlbyname)
sysctlbyname_will_always_fail_with_ENOSYS();
#else
sysctlbyname();
#endif

#if defined (__stub_stricmp) || defined (__stub___stricmp)
stricmp_will_always_fail_with_ENOSYS();
#else
stricmp();
#endif
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:118: warning: the `getwd' function is dangerous and should not be used.
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:82: undefined reference to `_sleep'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:112: undefined reference to `_getcwd'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:130: undefined reference to `_lseek'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:142: undefined reference to `_access'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:190: undefined reference to `_snprintf'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:220: undefined reference to `sbreak'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:262: undefined reference to `_set_output_format'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:268: undefined reference to `PXFGETARG'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:328: undefined reference to `_mkdir'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:340: undefined reference to `sysctlbyname'
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.c:346: undefined reference to `stricmp'
collect2: error: ld returned 1 exit status
Checking for functions [rand getdomainname realpath dlsym bzero uname usleep dlclose gethostname clock get_nprocs dlerror mkstemp fork getpagesize]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:17:6: warning: conflicting types for built-in function ???bzero??? [-Wbuiltin-declaration-mismatch]
 char bzero();
      ^~~~~
/tmp/petsc-VuReVk/config.functions/conftest.c:26:6: warning: conflicting types for built-in function ???fork??? [-Wbuiltin-declaration-mismatch]
 char fork();
      ^~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char rand();
char getdomainname();
char realpath();
char dlsym();
char bzero();
char uname();
char usleep();
char dlclose();
char gethostname();
char clock();
char get_nprocs();
char dlerror();
char mkstemp();
char fork();
char getpagesize();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_rand) || defined (__stub___rand)
rand_will_always_fail_with_ENOSYS();
#else
rand();
#endif

#if defined (__stub_getdomainname) || defined (__stub___getdomainname)
getdomainname_will_always_fail_with_ENOSYS();
#else
getdomainname();
#endif

#if defined (__stub_realpath) || defined (__stub___realpath)
realpath_will_always_fail_with_ENOSYS();
#else
realpath();
#endif

#if defined (__stub_dlsym) || defined (__stub___dlsym)
dlsym_will_always_fail_with_ENOSYS();
#else
dlsym();
#endif

#if defined (__stub_bzero) || defined (__stub___bzero)
bzero_will_always_fail_with_ENOSYS();
#else
bzero();
#endif

#if defined (__stub_uname) || defined (__stub___uname)
uname_will_always_fail_with_ENOSYS();
#else
uname();
#endif

#if defined (__stub_usleep) || defined (__stub___usleep)
usleep_will_always_fail_with_ENOSYS();
#else
usleep();
#endif

#if defined (__stub_dlclose) || defined (__stub___dlclose)
dlclose_will_always_fail_with_ENOSYS();
#else
dlclose();
#endif

#if defined (__stub_gethostname) || defined (__stub___gethostname)
gethostname_will_always_fail_with_ENOSYS();
#else
gethostname();
#endif

#if defined (__stub_clock) || defined (__stub___clock)
clock_will_always_fail_with_ENOSYS();
#else
clock();
#endif

#if defined (__stub_get_nprocs) || defined (__stub___get_nprocs)
get_nprocs_will_always_fail_with_ENOSYS();
#else
get_nprocs();
#endif

#if defined (__stub_dlerror) || defined (__stub___dlerror)
dlerror_will_always_fail_with_ENOSYS();
#else
dlerror();
#endif

#if defined (__stub_mkstemp) || defined (__stub___mkstemp)
mkstemp_will_always_fail_with_ENOSYS();
#else
mkstemp();
#endif

#if defined (__stub_fork) || defined (__stub___fork)
fork_will_always_fail_with_ENOSYS();
#else
fork();
#endif

#if defined (__stub_getpagesize) || defined (__stub___getpagesize)
getpagesize_will_always_fail_with_ENOSYS();
#else
getpagesize();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_RAND" to "1"
                      Defined "HAVE_GETDOMAINNAME" to "1"
                      Defined "HAVE_REALPATH" to "1"
                      Defined "HAVE_DLSYM" to "1"
                      Defined "HAVE_BZERO" to "1"
                      Defined "HAVE_UNAME" to "1"
                      Defined "HAVE_USLEEP" to "1"
                      Defined "HAVE_DLCLOSE" to "1"
                      Defined "HAVE_GETHOSTNAME" to "1"
                      Defined "HAVE_CLOCK" to "1"
                      Defined "HAVE_GET_NPROCS" to "1"
                      Defined "HAVE_DLERROR" to "1"
                      Defined "HAVE_MKSTEMP" to "1"
                      Defined "HAVE_FORK" to "1"
                      Defined "HAVE_GETPAGESIZE" to "1"
Checking for functions [memalign sigset gethostbyname gettimeofday readlink sigaction strcasecmp dlopen drand48 socket signal popen getrusage times time]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:19:6: warning: conflicting types for built-in function ???strcasecmp??? [-Wbuiltin-declaration-mismatch]
 char strcasecmp();
      ^~~~~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char memalign();
char sigset();
char gethostbyname();
char gettimeofday();
char readlink();
char sigaction();
char strcasecmp();
char dlopen();
char drand48();
char socket();
char signal();
char popen();
char getrusage();
char times();
char time();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_memalign) || defined (__stub___memalign)
memalign_will_always_fail_with_ENOSYS();
#else
memalign();
#endif

#if defined (__stub_sigset) || defined (__stub___sigset)
sigset_will_always_fail_with_ENOSYS();
#else
sigset();
#endif

#if defined (__stub_gethostbyname) || defined (__stub___gethostbyname)
gethostbyname_will_always_fail_with_ENOSYS();
#else
gethostbyname();
#endif

#if defined (__stub_gettimeofday) || defined (__stub___gettimeofday)
gettimeofday_will_always_fail_with_ENOSYS();
#else
gettimeofday();
#endif

#if defined (__stub_readlink) || defined (__stub___readlink)
readlink_will_always_fail_with_ENOSYS();
#else
readlink();
#endif

#if defined (__stub_sigaction) || defined (__stub___sigaction)
sigaction_will_always_fail_with_ENOSYS();
#else
sigaction();
#endif

#if defined (__stub_strcasecmp) || defined (__stub___strcasecmp)
strcasecmp_will_always_fail_with_ENOSYS();
#else
strcasecmp();
#endif

#if defined (__stub_dlopen) || defined (__stub___dlopen)
dlopen_will_always_fail_with_ENOSYS();
#else
dlopen();
#endif

#if defined (__stub_drand48) || defined (__stub___drand48)
drand48_will_always_fail_with_ENOSYS();
#else
drand48();
#endif

#if defined (__stub_socket) || defined (__stub___socket)
socket_will_always_fail_with_ENOSYS();
#else
socket();
#endif

#if defined (__stub_signal) || defined (__stub___signal)
signal_will_always_fail_with_ENOSYS();
#else
signal();
#endif

#if defined (__stub_popen) || defined (__stub___popen)
popen_will_always_fail_with_ENOSYS();
#else
popen();
#endif

#if defined (__stub_getrusage) || defined (__stub___getrusage)
getrusage_will_always_fail_with_ENOSYS();
#else
getrusage();
#endif

#if defined (__stub_times) || defined (__stub___times)
times_will_always_fail_with_ENOSYS();
#else
times();
#endif

#if defined (__stub_time) || defined (__stub___time)
time_will_always_fail_with_ENOSYS();
#else
time();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_MEMALIGN" to "1"
                      Defined "HAVE_SIGSET" to "1"
                      Defined "HAVE_GETHOSTBYNAME" to "1"
                      Defined "HAVE_GETTIMEOFDAY" to "1"
                      Defined "HAVE_READLINK" to "1"
                      Defined "HAVE_SIGACTION" to "1"
                      Defined "HAVE_STRCASECMP" to "1"
                      Defined "HAVE_DLOPEN" to "1"
                      Defined "HAVE_DRAND48" to "1"
                      Defined "HAVE_SOCKET" to "1"
                      Defined "HAVE_SIGNAL" to "1"
                      Defined "HAVE_POPEN" to "1"
                      Defined "HAVE_GETRUSAGE" to "1"
                      Defined "HAVE_TIMES" to "1"
                      Defined "HAVE_TIME" to "1"
Checking for functions [_sleep]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _sleep();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__sleep) || defined (__stub____sleep)
_sleep_will_always_fail_with_ENOSYS();
#else
_sleep();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_sleep'
collect2: error: ld returned 1 exit status
Checking for functions [snprintf]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:13:6: warning: conflicting types for built-in function ???snprintf??? [-Wbuiltin-declaration-mismatch]
 char snprintf();
      ^~~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char snprintf();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_snprintf) || defined (__stub___snprintf)
snprintf_will_always_fail_with_ENOSYS();
#else
snprintf();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_SNPRINTF" to "1"
Checking for functions [_getcwd]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _getcwd();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__getcwd) || defined (__stub____getcwd)
_getcwd_will_always_fail_with_ENOSYS();
#else
_getcwd();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_getcwd'
collect2: error: ld returned 1 exit status
Checking for functions [getwd]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char getwd();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_getwd) || defined (__stub___getwd)
getwd_will_always_fail_with_ENOSYS();
#else
getwd();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker:
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: warning: the `getwd' function is dangerous and should not be used.
                      Defined "HAVE_GETWD" to "1"
Checking for functions [_lseek]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _lseek();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__lseek) || defined (__stub____lseek)
_lseek_will_always_fail_with_ENOSYS();
#else
_lseek();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_lseek'
collect2: error: ld returned 1 exit status
Checking for functions [sleep]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char sleep();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_sleep) || defined (__stub___sleep)
sleep_will_always_fail_with_ENOSYS();
#else
sleep();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_SLEEP" to "1"
Checking for functions [_access]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _access();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__access) || defined (__stub____access)
_access_will_always_fail_with_ENOSYS();
#else
_access();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_access'
collect2: error: ld returned 1 exit status
Checking for functions [lseek]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char lseek();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_lseek) || defined (__stub___lseek)
lseek_will_always_fail_with_ENOSYS();
#else
lseek();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_LSEEK" to "1"
Checking for functions [access]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char access();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_access) || defined (__stub___access)
access_will_always_fail_with_ENOSYS();
#else
access();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_ACCESS" to "1"
Checking for functions [_snprintf]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _snprintf();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__snprintf) || defined (__stub____snprintf)
_snprintf_will_always_fail_with_ENOSYS();
#else
_snprintf();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_snprintf'
collect2: error: ld returned 1 exit status
Checking for functions [sbreak]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char sbreak();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_sbreak) || defined (__stub___sbreak)
sbreak_will_always_fail_with_ENOSYS();
#else
sbreak();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `sbreak'
collect2: error: ld returned 1 exit status
Checking for functions [getcwd]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char getcwd();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_getcwd) || defined (__stub___getcwd)
getcwd_will_always_fail_with_ENOSYS();
#else
getcwd();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
                      Defined "HAVE_GETCWD" to "1"
Checking for functions [_set_output_format]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _set_output_format();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__set_output_format) || defined (__stub____set_output_format)
_set_output_format_will_always_fail_with_ENOSYS();
#else
_set_output_format();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_set_output_format'
collect2: error: ld returned 1 exit status
Checking for functions [PXFGETARG]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char PXFGETARG();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_PXFGETARG) || defined (__stub___PXFGETARG)
PXFGETARG_will_always_fail_with_ENOSYS();
#else
PXFGETARG();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `PXFGETARG'
collect2: error: ld returned 1 exit status
Checking for functions [_mkdir]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _mkdir();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__mkdir) || defined (__stub____mkdir)
_mkdir_will_always_fail_with_ENOSYS();
#else
_mkdir();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `_mkdir'
collect2: error: ld returned 1 exit status
Checking for functions [sysctlbyname]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char sysctlbyname();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_sysctlbyname) || defined (__stub___sysctlbyname)
sysctlbyname_will_always_fail_with_ENOSYS();
#else
sysctlbyname();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `sysctlbyname'
collect2: error: ld returned 1 exit status
Checking for functions [stricmp]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char stricmp();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_stricmp) || defined (__stub___stricmp)
stricmp_will_always_fail_with_ENOSYS();
#else
stricmp();
#endif
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `stricmp'
collect2: error: ld returned 1 exit status
================================================================================
TEST checkMmap from config.functions(/home/ryan/projects/moose/petsc/config/BuildSystem/config/functions.py:214)
TESTING: checkMmap from config.functions(config/BuildSystem/config/functions.py:214)
  Check for functional mmap() to allocate shared memory and define HAVE_MMAP
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main() {
int fd;
 fd=open("/tmp/file",O_RDWR);
 mmap((void*)0,100,PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl 
              Defined "HAVE_MMAP" to "1"
================================================================================
TEST configureMemorySize from config.utilities.getResidentSetSize(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/getResidentSetSize.py:31)
TESTING: configureMemorySize from config.utilities.getResidentSetSize(config/BuildSystem/config/utilities/getResidentSetSize.py:31)
  Try to determine how to measure the memory usage
              Defined "USE_PROC_FOR_SIZE" to "1"
            Using /proc for PetscMemoryGetCurrentUsage()
================================================================================
TEST configureFortranCommandLine from config.utilities.fortranCommandLine(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/fortranCommandLine.py:27)
TESTING: configureFortranCommandLine from config.utilities.fortranCommandLine(config/BuildSystem/config/utilities/fortranCommandLine.py:27)
  Check for the mechanism to retrieve command line arguments in Fortran
              Defined "HAVE_FORTRAN_GET_COMMAND_ARGUMENT" to "1"
              Pushing language FC
              Checking for functions [] in library [''] []
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp  -g -O   /tmp/petsc-VuReVk/config.libraries/conftest.F90 
Successful compile:
Source:
      program main
      integer i
      character*(80) arg
       i = command_argument_count()
       call get_command_argument(i,arg)
      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.o   -lstdc++ -ldl 
                Popping language FC
              Popping language FC
              Pushing language C
              Defined "HAVE_GFORTRAN_IARGC" to "1"
              Popping language C
              Checking for functions [get_command_argument_] in library [''] ['-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char get_command_argument_();
static void _check_get_command_argument_() { get_command_argument_(); }

int main() {
_check_get_command_argument_();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_get_command_argument_':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `get_command_argument_'
collect2: error: ld returned 1 exit status
                Popping language C
              Checking for functions [getarg_] in library [''] ['-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char getarg_();
static void _check_getarg_() { getarg_(); }

int main() {
_check_getarg_();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_getarg_':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `getarg_'
collect2: error: ld returned 1 exit status
                Popping language C
                        Pushing language C
                        Popping language C
                        Pushing language C
                        Popping language C
                        Pushing language C
                        Popping language C
                        Pushing language C
                        Popping language C
                        Pushing language C
                        Popping language C
Checking for functions [ipxfargc_]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char ipxfargc_();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_ipxfargc_) || defined (__stub___ipxfargc_)
ipxfargc__will_always_fail_with_ENOSYS();
#else
ipxfargc_();
#endif
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `ipxfargc_'
collect2: error: ld returned 1 exit status
Checking for functions [f90_unix_MP_iargc]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char f90_unix_MP_iargc();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_f90_unix_MP_iargc) || defined (__stub___f90_unix_MP_iargc)
f90_unix_MP_iargc_will_always_fail_with_ENOSYS();
#else
f90_unix_MP_iargc();
#endif
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `f90_unix_MP_iargc'
collect2: error: ld returned 1 exit status
Checking for functions [PXFGETARG]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char PXFGETARG();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_PXFGETARG) || defined (__stub___PXFGETARG)
PXFGETARG_will_always_fail_with_ENOSYS();
#else
PXFGETARG();
#endif
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `PXFGETARG'
collect2: error: ld returned 1 exit status
Checking for functions [iargc_]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char iargc_();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_iargc_) || defined (__stub___iargc_)
iargc__will_always_fail_with_ENOSYS();
#else
iargc_();
#endif
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.functions/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.functions/conftest.c:24: undefined reference to `iargc_'
collect2: error: ld returned 1 exit status
Checking for functions [GETARG at 16]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.functions/conftest.c:13:12: error: stray ???@??? in program
 char GETARG at 16();
            ^
/tmp/petsc-VuReVk/config.functions/conftest.c:13:13: error: expected ???=???, ???,???, ???;???, ???asm??? or ???__attribute__??? before numeric constant
 char GETARG at 16();
             ^~
/tmp/petsc-VuReVk/config.functions/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.functions/conftest.c:21:27: error: missing ')' after "defined"
 #if defined (__stub_GETARG at 16) || defined (__stub___GETARG at 16)
                           ^
/tmp/petsc-VuReVk/config.functions/conftest.c:21:28: error: missing binary operator before token "16"
 #if defined (__stub_GETARG at 16) || defined (__stub___GETARG at 16)
                            ^~
/tmp/petsc-VuReVk/config.functions/conftest.c:24:7: error: stray ???@??? in program
 GETARG at 16();
       ^
/tmp/petsc-VuReVk/config.functions/conftest.c:24:1: error: ???GETARG??? undeclared (first use in this function)
 GETARG at 16();
 ^~~~~~
/tmp/petsc-VuReVk/config.functions/conftest.c:24:1: note: each undeclared identifier is reported only once for each function it appears in
/tmp/petsc-VuReVk/config.functions/conftest.c:24:7: error: expected ???;??? before numeric constant
 GETARG at 16();
       ^~~
       ;
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char GETARG at 16();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub_GETARG at 16) || defined (__stub___GETARG at 16)
GETARG at 16_will_always_fail_with_ENOSYS();
#else
GETARG at 16();
#endif
;
  return 0;
}
                  Compile failed inside link

Checking for functions [_gfortran_iargc]
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.functions/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.functions/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

/* System header to define __stub macros and hopefully no other prototypes since they would conflict with our 'char funcname()' declaration below. */
#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C" {
#endif

/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char _gfortran_iargc();

#ifdef __cplusplus
}
#endif

int main() {

#if defined (__stub__gfortran_iargc) || defined (__stub____gfortran_iargc)
_gfortran_iargc_will_always_fail_with_ENOSYS();
#else
_gfortran_iargc();
#endif
;
  return 0;
}
Executing: mpicc  -o /tmp/petsc-VuReVk/config.functions/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.functions/conftest.o -lstdc++ -ldl  -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread
                Defined "HAVE__GFORTRAN_IARGC" to "1"
================================================================================
TEST configureFeatureTestMacros from config.utilities.featureTestMacros(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/featureTestMacros.py:13)
TESTING: configureFeatureTestMacros from config.utilities.featureTestMacros(config/BuildSystem/config/utilities/featureTestMacros.py:13)
  Checks if certain feature test macros are support
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.utilities.featureTestMacros
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c:4:10: fatal error: sysctl.h: No such file or directory
 #include <sysctl.h>
          ^~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#define _POSIX_C_SOURCE 200112L
#include <sysctl.h>

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c 
Possible ERROR while running compiler:
stderr:
In file included from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:25,
                 from /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c:4:
/usr/include/features.h:184:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Wcpp]
 # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
   ^~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"
#define _BSD_SOURCE
#include<stdlib.h>

int main() {
;
  return 0;
}
              Defined "_BSD_SOURCE" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#define _DEFAULT_SOURCE
#include<stdlib.h>

int main() {
;
  return 0;
}
              Defined "_DEFAULT_SOURCE" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.featureTestMacros/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#define _GNU_SOURCE
#include <sched.h>

int main() {
cpu_set_t mset;
CPU_ZERO(&mset);;
  return 0;
}
              Defined "_GNU_SOURCE" to "1"
================================================================================
TEST configureMissingDefines from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:57)
TESTING: configureMissingDefines from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:57)
  Checks for limits
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.utilities.missing
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_LIMITS_H
  #include <limits.h>
#endif

int main() {
int i=INT_MAX;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_FLOAT_H
  #include <float.h>
#endif

int main() {
double d=DBL_MAX;

if (d);
;
  return 0;
}
================================================================================
TEST configureMissingUtypeTypedefs from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:67)
TESTING: configureMissingUtypeTypedefs from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:67)
   Checks if u_short is undefined 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sys/types.h>

int main() {
u_short foo;
;
  return 0;
}
================================================================================
TEST configureMissingFunctions from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:73)
TESTING: configureMissingFunctions from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:73)
  Checks for SOCKETS
================================================================================
TEST configureMissingSignals from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:93)
TESTING: configureMissingSignals from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:93)
  Check for missing signals, and define MISSING_<signal name> if necessary
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGABRT;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGALRM;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGBUS;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGCHLD;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGCONT;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGFPE;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGHUP;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGILL;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGINT;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGKILL;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGPIPE;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGQUIT;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGSEGV;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGSTOP;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGSYS;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGTERM;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGTRAP;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGTSTP;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGURG;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGUSR1;

if (i);
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <signal.h>

int main() {
int i=SIGUSR2;

if (i);
;
  return 0;
}
================================================================================
TEST configureMissingGetdomainnamePrototype from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:110)
TESTING: configureMissingGetdomainnamePrototype from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:110)
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#if !defined(_BSD_SOURCE)
#define _BSD_SOURCE
#endif

#if !defined(_DEFAULT_SOURCE)
#define _DEFAULT_SOURCE
#endif

#if !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif

#ifdef PETSC_HAVE_UNISTD_H
#include <unistd.h>
#endif
#ifdef PETSC_HAVE_NETDB_H
#include <netdb.h>
#endif

int main() {

int (*getdomainname_ptr)(char*,size_t) = getdomainname;
char test[10];
if (getdomainname_ptr(test,10)) return 1;
;
  return 0;
}
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.utilities.missing/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#if !defined(_BSD_SOURCE)
#define _BSD_SOURCE
#endif

#if !defined(_DEFAULT_SOURCE)
#define _DEFAULT_SOURCE
#endif

#if !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif

#ifdef PETSC_HAVE_UNISTD_H
#include <unistd.h>
#endif
#ifdef PETSC_HAVE_NETDB_H
#include <netdb.h>
#endif

int main() {

int (*getdomainname_ptr)(char*,size_t) = getdomainname;
char test[10];
if (getdomainname_ptr(test,10)) return 1;
;
  return 0;
}
                      Pushing language CXX
                      Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.utilities.missing/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -lstdc++ -ldl 
              Popping language Cxx
================================================================================
TEST configureMissingSrandPrototype from config.utilities.missing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/missing.py:135)
TESTING: configureMissingSrandPrototype from config.utilities.missing(config/BuildSystem/config/utilities/missing.py:135)
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#if !defined(_BSD_SOURCE)
#define _BSD_SOURCE
#endif

#if !defined(_DEFAULT_SOURCE)
#define _DEFAULT_SOURCE
#endif

#if !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif

#ifdef PETSC_HAVE_STDLIB_H
#include <stdlib.h>
#endif

int main() {

double (*drand48_ptr)(void) = drand48;
void (*srand48_ptr)(long int) = srand48;
long int seed=10;
srand48_ptr(seed);
if (drand48_ptr() > 0.5) return 1;
;
  return 0;
}
              Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.missing -fPIC -fopenmp -g -O       /tmp/petsc-VuReVk/config.utilities.missing/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#if !defined(_BSD_SOURCE)
#define _BSD_SOURCE
#endif

#if !defined(_DEFAULT_SOURCE)
#define _DEFAULT_SOURCE
#endif

#if !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif

#ifdef PETSC_HAVE_STDLIB_H
#include <stdlib.h>
#endif

int main() {

double (*drand48_ptr)(void) = drand48;
void (*srand48_ptr)(long int) = srand48;
long int seed=10;
srand48_ptr(seed);
if (drand48_ptr() > 0.5) return 1;
;
  return 0;
}
                      Pushing language CXX
                      Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.utilities.missing/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.utilities.missing/conftest.o -lstdc++ -ldl 
              Popping language Cxx
================================================================================
TEST configureFPTrap from config.utilities.FPTrap(/home/ryan/projects/moose/petsc/config/BuildSystem/config/utilities/FPTrap.py:28)
TESTING: configureFPTrap from config.utilities.FPTrap(config/BuildSystem/config/utilities/FPTrap.py:28)
  Checking the handling of floating point traps
Checking for header: sigfpe.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sigfpe.h: No such file or directory
 #include <sigfpe.h>
          ^~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sigfpe.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sigfpe.h: No such file or directory
 #include <sigfpe.h>
          ^~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: sigfpe.h: No such file or directory #include <sigfpe.h>          ^~~~~~~~~~compilation terminated.:
Checking for header: fpxcp.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: fpxcp.h: No such file or directory
 #include <fpxcp.h>
          ^~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <fpxcp.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: fpxcp.h: No such file or directory
 #include <fpxcp.h>
          ^~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: fpxcp.h: No such file or directory #include <fpxcp.h>          ^~~~~~~~~compilation terminated.:
Checking for header: floatingpoint.h
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Possible ERROR while running preprocessor: exit code 256
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2stderr:
/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: floatingpoint.h: No such file or directory
 #include <floatingpoint.h>
          ^~~~~~~~~~~~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
#include <floatingpoint.h>
Preprocess stderr before filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: floatingpoint.h: No such file or directory
 #include <floatingpoint.h>
          ^~~~~~~~~~~~~~~~~
compilation terminated.
:
Preprocess stderr after filtering:/tmp/petsc-VuReVk/config.headers/conftest.c:3:10: fatal error: floatingpoint.h: No such file or directory #include <floatingpoint.h>          ^~~~~~~~~~~~~~~~~compilation terminated.:
================================================================================
TEST configureScalarType from PETSc.options.scalarTypes(/home/ryan/projects/moose/petsc/config/PETSc/options/scalarTypes.py:37)
TESTING: configureScalarType from PETSc.options.scalarTypes(config/PETSc/options/scalarTypes.py:37)
  Choose between real and complex numbers
              Defined "USE_SCALAR_REAL" to "1"
            Scalar type is real
              Pushing language C
                          All intermediate test results are stored in /tmp/petsc-VuReVk/PETSc.options.scalarTypes
Executing: mpicc -c -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <math.h>

int main() {
double b = 2.0; int a = isnormal(b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -lstdc++ -ldl 
              Defined "HAVE_ISNORMAL" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <math.h>

int main() {
double b = 2.0; int a = isnan(b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -lstdc++ -ldl 
              Defined "HAVE_ISNAN" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <math.h>

int main() {
double b = 2.0; int a = isinf(b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -lstdc++ -ldl 
              Defined "HAVE_ISINF" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c:6:24: warning: implicit declaration of function ???_isnan??? [-Wimplicit-function-declaration]
 double b = 2.0;int a = _isnan(b);
                        ^~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"
#include <float.h>

int main() {
double b = 2.0;int a = _isnan(b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o: in function `main':
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c:6: undefined reference to `_isnan'
collect2: error: ld returned 1 exit status
Executing: mpicc -c -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c:6:24: warning: implicit declaration of function ???_finite??? [-Wimplicit-function-declaration]
 double b = 2.0;int a = _finite(b);
                        ^~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"
#include <float.h>

int main() {
double b = 2.0;int a = _finite(b);
;
  return 0;
}
                      Pushing language C
                      Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.o: in function `main':
/tmp/petsc-VuReVk/PETSc.options.scalarTypes/conftest.c:6: undefined reference to `_finite'
collect2: error: ld returned 1 exit status
              Popping language C
================================================================================
TEST configurePrecision from PETSc.options.scalarTypes(/home/ryan/projects/moose/petsc/config/PETSc/options/scalarTypes.py:79)
TESTING: configurePrecision from PETSc.options.scalarTypes(config/PETSc/options/scalarTypes.py:79)
  Set the default real number precision for PETSc objects
              Defined "USE_REAL_DOUBLE" to "1"
              Defined make macro "PETSC_SCALAR_SIZE" to "64"
            Precision is double
================================================================================
TEST configureMkdir from config.programs(/home/ryan/projects/moose/petsc/config/BuildSystem/config/programs.py:23)
TESTING: configureMkdir from config.programs(config/BuildSystem/config/programs.py:23)
  Make sure we can have mkdir automatically make intermediate directories
Checking for program /home/ryan/bin/mkdir...not found
Checking for program /usr/local/bin/mkdir...not found
Checking for program /home/ryan/bin/mkdir...not found
Checking for program /usr/local/bin/mkdir...not found
Checking for program /usr/local/sbin/mkdir...not found
Checking for program /usr/bin/mkdir...found
Executing: /usr/bin/mkdir -p .conftest/tmp
            Adding -p flag to /usr/bin/mkdir -p to automatically create directories
              Defined make macro "MKDIR" to "/usr/bin/mkdir -p"
================================================================================
TEST configureAutoreconf from config.programs(/home/ryan/projects/moose/petsc/config/BuildSystem/config/programs.py:45)
TESTING: configureAutoreconf from config.programs(config/BuildSystem/config/programs.py:45)
  Check for autoreconf
Checking for program /home/ryan/bin/autoreconf...not found
Checking for program /usr/local/bin/autoreconf...not found
Checking for program /home/ryan/bin/autoreconf...not found
Checking for program /usr/local/bin/autoreconf...not found
Checking for program /usr/local/sbin/autoreconf...not found
Checking for program /usr/bin/autoreconf...found
              All intermediate test results are stored in /tmp/petsc-VuReVk/config.programs
Executing: cd /tmp/petsc-VuReVk/config.programs/autoconfdir&&/usr/bin/autoreconf
            autoreconf test successful!
Checking for program /home/ryan/bin/libtoolize...not found
Checking for program /usr/local/bin/libtoolize...not found
Checking for program /home/ryan/bin/libtoolize...not found
Checking for program /usr/local/bin/libtoolize...not found
Checking for program /usr/local/sbin/libtoolize...not found
Checking for program /usr/bin/libtoolize...found
================================================================================
TEST configurePrograms from config.programs(/home/ryan/projects/moose/petsc/config/BuildSystem/config/programs.py:72)
TESTING: configurePrograms from config.programs(config/BuildSystem/config/programs.py:72)
  Check for the programs needed to build and run PETSc
Checking for program /home/ryan/bin/sh...not found
Checking for program /usr/local/bin/sh...not found
Checking for program /home/ryan/bin/sh...not found
Checking for program /usr/local/bin/sh...not found
Checking for program /usr/local/sbin/sh...not found
Checking for program /usr/bin/sh...found
                Defined make macro "SHELL" to "/usr/bin/sh"
Checking for program /home/ryan/bin/sed...not found
Checking for program /usr/local/bin/sed...not found
Checking for program /home/ryan/bin/sed...not found
Checking for program /usr/local/bin/sed...not found
Checking for program /usr/local/sbin/sed...not found
Checking for program /usr/bin/sed...found
                Defined make macro "SED" to "/usr/bin/sed"
Executing: /usr/bin/sed -i s/sed/sd/g "/tmp/petsc-VuReVk/config.programs/sed1"
            Adding SEDINPLACE cmd: /usr/bin/sed -i
              Defined make macro "SEDINPLACE" to "/usr/bin/sed -i"
Checking for program /home/ryan/bin/mv...not found
Checking for program /usr/local/bin/mv...not found
Checking for program /home/ryan/bin/mv...not found
Checking for program /usr/local/bin/mv...not found
Checking for program /usr/local/sbin/mv...not found
Checking for program /usr/bin/mv...found
                Defined make macro "MV" to "/usr/bin/mv"
Checking for program /home/ryan/bin/cp...not found
Checking for program /usr/local/bin/cp...not found
Checking for program /home/ryan/bin/cp...not found
Checking for program /usr/local/bin/cp...not found
Checking for program /usr/local/sbin/cp...not found
Checking for program /usr/bin/cp...found
                Defined make macro "CP" to "/usr/bin/cp"
Checking for program /home/ryan/bin/grep...not found
Checking for program /usr/local/bin/grep...not found
Checking for program /home/ryan/bin/grep...not found
Checking for program /usr/local/bin/grep...not found
Checking for program /usr/local/sbin/grep...not found
Checking for program /usr/bin/grep...found
                Defined make macro "GREP" to "/usr/bin/grep"
Checking for program /home/ryan/bin/rm...not found
Checking for program /usr/local/bin/rm...not found
Checking for program /home/ryan/bin/rm...not found
Checking for program /usr/local/bin/rm...not found
Checking for program /usr/local/sbin/rm...not found
Checking for program /usr/bin/rm...found
                Defined make macro "RM" to "/usr/bin/rm -f"
Checking for program /home/ryan/bin/diff...not found
Checking for program /usr/local/bin/diff...not found
Checking for program /home/ryan/bin/diff...not found
Checking for program /usr/local/bin/diff...not found
Checking for program /usr/local/sbin/diff...not found
Checking for program /usr/bin/diff...found
Executing: "/usr/bin/diff" -w "/tmp/petsc-VuReVk/config.programs/diff1" "/tmp/petsc-VuReVk/config.programs/diff2"
              Defined make macro "DIFF" to "/usr/bin/diff -w"
Checking for program /usr/ucb/ps...not found
Checking for program /usr/usb/ps...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/ps...not found
Checking for program /home/ryan/bin/gzip...not found
Checking for program /usr/local/bin/gzip...not found
Checking for program /home/ryan/bin/gzip...not found
Checking for program /usr/local/bin/gzip...not found
Checking for program /usr/local/sbin/gzip...not found
Checking for program /usr/bin/gzip...found
                Defined make macro "GZIP" to "/usr/bin/gzip"
              Defined "HAVE_GZIP" to "1"
              Defined make macro "PYTHON" to "/usr/bin/python2"
Checking for program /home/ryan/bin/m4...not found
Checking for program /usr/local/bin/m4...not found
Checking for program /home/ryan/bin/m4...not found
Checking for program /usr/local/bin/m4...not found
Checking for program /usr/local/sbin/m4...not found
Checking for program /usr/bin/m4...found
                Defined make macro "M4" to "/usr/bin/m4"
================================================================================
TEST configureMake from config.packages.make(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/make.py:85)
TESTING: configureMake from config.packages.make(config/BuildSystem/config/packages/make.py:85)
  Check Guesses for GNU make
Executing: gmake --version
GNUMake check failed: Could not execute "gmake --version":
/bin/sh: gmake: command not found

Executing: make --version
stdout:
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Checking for program /home/ryan/bin/make...not found
Checking for program /usr/local/bin/make...not found
Checking for program /home/ryan/bin/make...not found
Checking for program /usr/local/bin/make...not found
Checking for program /usr/local/sbin/make...not found
Checking for program /usr/bin/make...found
                Defined make macro "MAKE" to "/usr/bin/make"
================================================================================
TEST setupGNUMake from config.packages.make(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/make.py:113)
TESTING: setupGNUMake from config.packages.make(config/BuildSystem/config/packages/make.py:113)
  Setup other GNU make stuff
Executing: uname -s
stdout: Linux
Executing: uname -s
stdout: Linux
              Defined make macro "MAKE_IS_GNUMAKE" to "1"
              Defined make rule "libc" with dependencies "${LIBNAME}(${OBJSC})" and code []
              Defined make rule "libcxx" with dependencies "${LIBNAME}(${OBJSCXX})" and code []
              Defined make rule "libcu" with dependencies "${LIBNAME}(${OBJSCU})" and code []
              Defined make rule "libf" with dependencies "${OBJSF}" and code -${AR} ${AR_FLAGS} ${LIBNAME} ${OBJSF}
              Defined make macro "OMAKE_PRINTDIR " to "/usr/bin/make  --print-directory"
              Defined make macro "OMAKE" to "/usr/bin/make  --no-print-directory"
              Defined make macro "MAKE_PAR_OUT_FLG" to "--output-sync=recurse"
================================================================================
TEST configureMakeNP from config.packages.make(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/make.py:161)
TESTING: configureMakeNP from config.packages.make(config/BuildSystem/config/packages/make.py:161)
  check no of cores on the build machine [perhaps to do make '-j ncores']
            module multiprocessing found 4 cores: using make_np = 4
              Defined make macro "MAKE_NP" to "4"
              Defined make macro "MAKE_TEST_NP" to "2"
              Defined make macro "MAKE_LOAD" to "6.0"
              Defined make macro "NPMAX" to "4"
================================================================================
TEST alternateConfigureLibrary from config.packages.OpenMPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.OpenMPI(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
Executing: uname -s
stdout: Linux
Executing: uname -s
stdout: Linux
================================================================================
TEST alternateConfigureLibrary from config.packages.MPICH(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.MPICH(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.MPI(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:524)
TESTING: configureLibrary from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:524)
  Calls the regular package configureLibrary and then does an additional test needed by MPI
==================================================================================
              Checking for a functional MPI
              Checking for library in Compiler specific search MPI: []
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                  Checking for functions [MPI_Init MPI_Comm_create] in library [] []
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_Init();
static void _check_MPI_Init() { MPI_Init(); }
char MPI_Comm_create();
static void _check_MPI_Comm_create() { MPI_Comm_create(); }

int main() {
_check_MPI_Init();
_check_MPI_Comm_create();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                    Popping language C
                  No functions to check for in library [] []
              Checking for headers Compiler specific search MPI: ['/usr/include', '/usr/lib/openmpi']
                  Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['mpi.h'] in ['/usr/include', '/usr/lib/openmpi']
                    Checking include with compiler flags var CPPFLAGS ['/usr/include', '/usr/lib/openmpi']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.headers  -I/usr/include -I/usr/lib/openmpi /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/mpi.h" 1 3 4
# 232 "/usr/include/mpi.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long int ptrdiff_t;
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef struct {
  long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
  long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
# 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
} max_align_t;
# 233 "/usr/include/mpi.h" 2 3 4
# 265 "/usr/include/mpi.h" 3 4
# 1 "/usr/include/mpi_portable_platform.h" 1 3 4
# 266 "/usr/include/mpi.h" 2 3 4
# 342 "/usr/include/mpi.h" 3 4
typedef ptrdiff_t MPI_Aint;
typedef long long MPI_Offset;
typedef long long MPI_Count;
typedef struct ompi_communicator_t *MPI_Comm;
typedef struct ompi_datatype_t *MPI_Datatype;
typedef struct ompi_errhandler_t *MPI_Errhandler;
typedef struct ompi_file_t *MPI_File;
typedef struct ompi_group_t *MPI_Group;
typedef struct ompi_info_t *MPI_Info;
typedef struct ompi_op_t *MPI_Op;
typedef struct ompi_request_t *MPI_Request;
typedef struct ompi_message_t *MPI_Message;
typedef struct ompi_status_public_t MPI_Status;
typedef struct ompi_win_t *MPI_Win;
typedef struct mca_base_var_enum_t *MPI_T_enum;
typedef struct ompi_mpit_cvar_handle_t *MPI_T_cvar_handle;
typedef struct mca_base_pvar_handle_t *MPI_T_pvar_handle;
typedef struct mca_base_pvar_session_t *MPI_T_pvar_session;
struct ompi_status_public_t {
    int MPI_SOURCE;
    int MPI_TAG;
    int MPI_ERROR;
    int _cancelled;
    size_t _ucount;
};
typedef struct ompi_status_public_t ompi_status_public_t;
typedef int (MPI_Datarep_extent_function)(MPI_Datatype, MPI_Aint *, void *);
typedef int (MPI_Datarep_conversion_function)(void *, MPI_Datatype,
                                              int, void *, MPI_Offset, void *);
typedef void (MPI_Comm_errhandler_function)(MPI_Comm *, int *, ...);
typedef void (ompi_file_errhandler_fn)(MPI_File *, int *, ...);
typedef void (MPI_Win_errhandler_function)(MPI_Win *, int *, ...);
typedef void (MPI_User_function)(void *, void *, int *, MPI_Datatype *);
typedef int (MPI_Comm_copy_attr_function)(MPI_Comm, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Comm_delete_attr_function)(MPI_Comm, int, void *, void *);
typedef int (MPI_Type_copy_attr_function)(MPI_Datatype, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Type_delete_attr_function)(MPI_Datatype, int,
                                              void *, void *);
typedef int (MPI_Win_copy_attr_function)(MPI_Win, int, void *,
                                           void *, void *, int *);
typedef int (MPI_Win_delete_attr_function)(MPI_Win, int, void *, void *);
typedef int (MPI_Grequest_query_function)(void *, MPI_Status *);
typedef int (MPI_Grequest_free_function)(void *);
typedef int (MPI_Grequest_cancel_function)(void *, int);
typedef MPI_Comm_errhandler_function MPI_Comm_errhandler_fn
        __attribute__((__deprecated__("MPI_Comm_errhandler_fn was deprecated in MPI-2.2; use MPI_Comm_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_fn
        __attribute__((__deprecated__("MPI_File_errhandler_fn was deprecated in MPI-2.2; use MPI_File_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_function;
typedef MPI_Win_errhandler_function MPI_Win_errhandler_fn
        __attribute__((__deprecated__("MPI_Win_errhandler_fn was deprecated in MPI-2.2; use MPI_Win_errhandler_function instead")));
# 509 "/usr/include/mpi.h" 3 4
enum {
    MPI_TAG_UB,
    MPI_HOST,
    MPI_IO,
    MPI_WTIME_IS_GLOBAL,
    MPI_APPNUM,
    MPI_LASTUSEDCODE,
    MPI_UNIVERSE_SIZE,
    MPI_WIN_BASE,
    MPI_WIN_SIZE,
    MPI_WIN_DISP_UNIT,
    MPI_WIN_CREATE_FLAVOR,
    MPI_WIN_MODEL,
    IMPI_CLIENT_SIZE,
    IMPI_CLIENT_COLOR,
    IMPI_HOST_SIZE,
    IMPI_HOST_COLOR
};
# 623 "/usr/include/mpi.h" 3 4
enum {
  MPI_IDENT,
  MPI_CONGRUENT,
  MPI_SIMILAR,
  MPI_UNEQUAL
};
enum {
  MPI_THREAD_SINGLE,
  MPI_THREAD_FUNNELED,
  MPI_THREAD_SERIALIZED,
  MPI_THREAD_MULTIPLE
};
enum {
  MPI_COMBINER_NAMED,
  MPI_COMBINER_DUP,
  MPI_COMBINER_CONTIGUOUS,
  MPI_COMBINER_VECTOR,
  MPI_COMBINER_HVECTOR_INTEGER,
  MPI_COMBINER_HVECTOR,
  MPI_COMBINER_INDEXED,
  MPI_COMBINER_HINDEXED_INTEGER,
  MPI_COMBINER_HINDEXED,
  MPI_COMBINER_INDEXED_BLOCK,
  MPI_COMBINER_STRUCT_INTEGER,
  MPI_COMBINER_STRUCT,
  MPI_COMBINER_SUBARRAY,
  MPI_COMBINER_DARRAY,
  MPI_COMBINER_F90_REAL,
  MPI_COMBINER_F90_COMPLEX,
  MPI_COMBINER_F90_INTEGER,
  MPI_COMBINER_RESIZED,
  MPI_COMBINER_HINDEXED_BLOCK
};
enum {
  MPI_COMM_TYPE_SHARED,
  OMPI_COMM_TYPE_HWTHREAD,
  OMPI_COMM_TYPE_CORE,
  OMPI_COMM_TYPE_L1CACHE,
  OMPI_COMM_TYPE_L2CACHE,
  OMPI_COMM_TYPE_L3CACHE,
  OMPI_COMM_TYPE_SOCKET,
  OMPI_COMM_TYPE_NUMA,
  OMPI_COMM_TYPE_BOARD,
  OMPI_COMM_TYPE_HOST,
  OMPI_COMM_TYPE_CU,
  OMPI_COMM_TYPE_CLUSTER
};
enum {
  MPI_T_VERBOSITY_USER_BASIC,
  MPI_T_VERBOSITY_USER_DETAIL,
  MPI_T_VERBOSITY_USER_ALL,
  MPI_T_VERBOSITY_TUNER_BASIC,
  MPI_T_VERBOSITY_TUNER_DETAIL,
  MPI_T_VERBOSITY_TUNER_ALL,
  MPI_T_VERBOSITY_MPIDEV_BASIC,
  MPI_T_VERBOSITY_MPIDEV_DETAIL,
  MPI_T_VERBOSITY_MPIDEV_ALL
};
enum {
  MPI_T_SCOPE_CONSTANT,
  MPI_T_SCOPE_READONLY,
  MPI_T_SCOPE_LOCAL,
  MPI_T_SCOPE_GROUP,
  MPI_T_SCOPE_GROUP_EQ,
  MPI_T_SCOPE_ALL,
  MPI_T_SCOPE_ALL_EQ
};
enum {
  MPI_T_BIND_NO_OBJECT,
  MPI_T_BIND_MPI_COMM,
  MPI_T_BIND_MPI_DATATYPE,
  MPI_T_BIND_MPI_ERRHANDLER,
  MPI_T_BIND_MPI_FILE,
  MPI_T_BIND_MPI_GROUP,
  MPI_T_BIND_MPI_OP,
  MPI_T_BIND_MPI_REQUEST,
  MPI_T_BIND_MPI_WIN,
  MPI_T_BIND_MPI_MESSAGE,
  MPI_T_BIND_MPI_INFO
};
enum {
  MPI_T_PVAR_CLASS_STATE,
  MPI_T_PVAR_CLASS_LEVEL,
  MPI_T_PVAR_CLASS_SIZE,
  MPI_T_PVAR_CLASS_PERCENTAGE,
  MPI_T_PVAR_CLASS_HIGHWATERMARK,
  MPI_T_PVAR_CLASS_LOWWATERMARK,
  MPI_T_PVAR_CLASS_COUNTER,
  MPI_T_PVAR_CLASS_AGGREGATE,
  MPI_T_PVAR_CLASS_TIMER,
  MPI_T_PVAR_CLASS_GENERIC
};
# 820 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_DELETE_FN( MPI_Datatype datatype,
                                                  int type_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_COPY_FN( MPI_Datatype datatype,
                                                int type_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_DUP_FN( MPI_Datatype datatype,
                                          int type_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_DELETE_FN( MPI_Comm comm,
                                                  int comm_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_COPY_FN( MPI_Comm comm,
                                                int comm_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_DUP_FN( MPI_Comm comm, int comm_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_DELETE_FN( MPI_Win window,
                                                 int win_keyval,
                                                 void* attribute_val_out,
                                                 void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_COPY_FN( MPI_Win window, int win_keyval,
                                               void* extra_state,
                                               void* attribute_val_in,
                                               void* attribute_val_out,
                                               int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_DUP_FN( MPI_Win window, int win_keyval,
                                         void* extra_state,
                                         void* attribute_val_in,
                                         void* attribute_val_out,
                                         int* flag );
# 874 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_world;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_self;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_empty;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_request_t ompi_request_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_no_proc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_min;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_max;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_sum;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_prod;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_land;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_band;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_maxloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_minloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_replace;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_no_op;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_datatype_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_signed_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_byte;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_wchar;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_packed;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_character;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_longdbl_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex32;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_aint;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_offset;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_count;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_float_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_long_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errhandler_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_are_fatal;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_return;
__attribute__((visibility("default"))) extern struct ompi_predefined_win_t ompi_mpi_win_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_file_t ompi_mpi_file_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_env;
__attribute__((visibility("default"))) extern int *MPI_F_STATUS_IGNORE;
__attribute__((visibility("default"))) extern int *MPI_F_STATUSES_IGNORE;
# 1185 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int MPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                  int target_rank, MPI_Aint target_disp, int target_count,
                                  MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int MPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int MPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int MPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                 void *baseptr);
__attribute__((visibility("default"))) int MPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                             int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                             int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int MPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int MPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                   const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int MPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                int periods[], int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                const int periods[], int *newrank);
__attribute__((visibility("default"))) int MPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int MPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                  int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int MPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int MPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int MPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int MPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                   MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int MPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                          MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                          int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm MPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int MPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                         const int degrees[], const int targets[],
                                         const int weights[], MPI_Info info,
                                         int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int MPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                  int indegree, const int sources[],
                                                  const int sourceweights[],
                                                  int outdegree,
                                                  const int destinations[],
                                                  const int destweights[],
                                                  MPI_Info info, int reorder,
                                                  MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                           int sources[], int sourceweights[],
                                           int maxoutdegree,
                                           int destinations[],
                                           int destweights[]);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                  int *inneighbors,
                                                  int *outneighbors,
                                                  int *weighted);
__attribute__((visibility("default"))) int MPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int MPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int MPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int MPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int MPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int MPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                  int root, MPI_Comm comm, MPI_Comm *intercomm,
                                  int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                           const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                           int root, MPI_Comm comm, MPI_Comm *intercomm,
                                           int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int MPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                        void *result_addr, MPI_Datatype datatype, int target_rank,
                                        MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int MPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler MPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int MPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int MPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int MPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                    int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File MPI_File_f2c(int file);
__attribute__((visibility("default"))) int MPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int MPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                 MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int MPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int MPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                     MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                     MPI_Datatype *etype,
                                     MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int MPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                    int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                        int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read(MPI_File fh, void *buf, int count,
                                 MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all(MPI_File fh, void *buf, int count,
                                     MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write(MPI_File fh, const void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all(MPI_File fh, const void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_all(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                            MPI_Offset *disp);
__attribute__((visibility("default"))) int MPI_File_read_shared(MPI_File fh, void *buf, int count,
                                        MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_shared(MPI_File fh, const void *buf, int count,
      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                              int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                           MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                               MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                            MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int MPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int MPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int MPI_Finalize(void);
__attribute__((visibility("default"))) int MPI_Finalized(int *flag);
__attribute__((visibility("default"))) int MPI_Free_mem(void *base);
__attribute__((visibility("default"))) int MPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int MPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype, MPI_Count *count);
__attribute__((visibility("default"))) int MPI_Get(void *origin_addr, int origin_count,
                           MPI_Datatype origin_datatype, int target_rank,
                           MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                      void *result_addr, int result_count, MPI_Datatype result_datatype,
                                      int target_rank, MPI_Aint target_disp, int target_count,
                                      MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int MPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                    const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int MPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                 int index[], int edges[]);
__attribute__((visibility("default"))) int MPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                 int *newrank);
__attribute__((visibility("default"))) int MPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int MPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                       int neighbors[]);
__attribute__((visibility("default"))) int MPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int MPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int MPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                      MPI_Grequest_free_function *free_fn,
                                      MPI_Grequest_cancel_function *cancel_fn,
                                      void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int MPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int MPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group MPI_Group_f2c(int group);
__attribute__((visibility("default"))) int MPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                          MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int MPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int MPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                             MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int MPI_Group_union(MPI_Group group1, MPI_Group group2,
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Improbe(int source, int tag, MPI_Comm comm,
                               int *flag, MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Imrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int MPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int MPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info MPI_Info_f2c(int info);
__attribute__((visibility("default"))) int MPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                char *value, int *flag);
__attribute__((visibility("default"))) int MPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int MPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int MPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                         int *flag);
__attribute__((visibility("default"))) int MPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int MPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int MPI_Initialized(int *flag);
__attribute__((visibility("default"))) int MPI_Init_thread(int *argc, char ***argv, int required,
                                   int *provided);
__attribute__((visibility("default"))) int MPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                        MPI_Comm bridge_comm, int remote_leader,
                                        int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                       MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                              MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int MPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int MPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message MPI_Message_f2c(int message);
__attribute__((visibility("default"))) int MPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Mrecv(void *buf, int count, MPI_Datatype type,
                             MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int displs[],
                                           MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                         void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                         MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                          void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                          void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int MPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int MPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op MPI_Op_f2c(int op);
__attribute__((visibility("default"))) int MPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                     MPI_Datatype datatype, void *outbuf,
                                     MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int MPI_Pack_external_size(const char datarep[], int incount,
                                          MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int MPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                            void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                 int *size);
__attribute__((visibility("default"))) int MPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Publish_name(const char *service_name, MPI_Info info,
                                    const char *port_name);
__attribute__((visibility("default"))) int MPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                           int target_rank, MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int MPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                 int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                            int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                    MPI_Datatype datatype, MPI_Op op);
__attribute__((visibility("default"))) int MPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Register_datarep(const char *datarep,
                                        MPI_Datarep_conversion_function *read_conversion_fn,
                                        MPI_Datarep_conversion_function *write_conversion_fn,
                                        MPI_Datarep_extent_function *dtype_file_extent_fn,
                                        void *extra_state);
__attribute__((visibility("default"))) int MPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request MPI_Request_f2c(int request);
__attribute__((visibility("default"))) int MPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Request_get_status(MPI_Request request, int *flag,
                                          MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                            MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op,
                                       MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_cout,
                            MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                 int dest, int tag, MPI_Comm comm,
                                 MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                            int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                int dest, int sendtag, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int source, int recvtag,
                                MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                        int dest, int sendtag, int source, int recvtag,
                                        MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int MPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int MPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int MPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int MPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                           int count);
__attribute__((visibility("default"))) int MPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                             MPI_Count count);
__attribute__((visibility("default"))) int MPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                               MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Testany(int count, MPI_Request array_of_requests[], int *index,
                               int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int MPI_Testsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int MPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                       MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_darray(int size, int rank, int ndims,
                                          const int gsize_array[], const int distrib_array[],
                                          const int darg_array[], const int psize_array[],
                                          int order, MPI_Datatype oldtype,
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed_block(int count, int blocklength,
                                                  const MPI_Aint array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                            const MPI_Aint array_of_displacements[],
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                           MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                          MPI_Type_delete_attr_function *type_delete_attr_fn,
                                          int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Type_create_indexed_block(int count, int blocklength,
                                                 const int array_of_displacements[],
                                                 MPI_Datatype oldtype,
                                                 MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                          const MPI_Aint array_of_displacements[],
                                          const MPI_Datatype array_of_types[],
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                            const int start_array[], int order,
                                            MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                           MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int MPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype MPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int MPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                         int max_addresses, int max_datatypes,
                                         int array_of_integers[],
                                         MPI_Aint array_of_addresses[],
                                         MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int MPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                         int *num_addresses, int *num_datatypes,
                                         int *combiner);
__attribute__((visibility("default"))) int MPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                       MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                         MPI_Count *extent);
__attribute__((visibility("default"))) int MPI_Type_get_name(MPI_Datatype type, char *type_name,
                                     int *resultlen);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                            MPI_Aint *true_extent);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                              MPI_Count *true_extent);
__attribute__((visibility("default"))) int MPI_Type_indexed(int count, const int array_of_blocklengths[],
                                    const int array_of_displacements[],
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                     void *attr_val);
__attribute__((visibility("default"))) int MPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int MPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int MPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int MPI_Type_vector(int count, int blocklength, int stride,
                                   MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Unpack(const void *inbuf, int insize, int *position,
                              void *outbuf, int outcount, MPI_Datatype datatype,
                              MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Unpublish_name(const char *service_name, MPI_Info info, const char *port_name);
__attribute__((visibility("default"))) int MPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                        MPI_Aint *position, void *outbuf, int outcount,
                                        MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Waitall(int count, MPI_Request array_of_requests[],
                               MPI_Status *array_of_statuses);
__attribute__((visibility("default"))) int MPI_Waitany(int count, MPI_Request array_of_requests[],
                               int *index, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                    MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                           MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int MPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int MPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                  MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                             MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                         MPI_Win_delete_attr_function *win_delete_attr_fn,
                                         int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int MPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win MPI_Win_f2c(int win);
__attribute__((visibility("default"))) int MPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int MPI_Win_get_attr(MPI_Win win, int win_keyval,
                                    void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int MPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int MPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int MPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double MPI_Wtick(void);
__attribute__((visibility("default"))) double MPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int PMPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int PMPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int PMPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                  void *baseptr);
__attribute__((visibility("default"))) int PMPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                          const int degrees[], const int targets[],
                                          const int weights[], MPI_Info info,
                                          int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                   int indegree, const int sources[],
                                                   const int sourceweights[],
                                                   int outdegree,
                                                   const int destinations[],
                                                   const int destweights[],
                                                   MPI_Info info, int reorder,
                                                   MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                            int sources[], int sourceweights[],
                                            int maxoutdegree,
                                             int destinations[],
                                            int destweights[]);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                   int *inneighbors,
                                                   int *outneighbors,
                                                   int *weighted);
__attribute__((visibility("default"))) int PMPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                              int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int PMPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int PMPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                    const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int PMPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                 int periods[], int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                 const int periods[], int *newrank);
__attribute__((visibility("default"))) int PMPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int PMPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                   int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int PMPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int PMPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int PMPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int PMPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int PMPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                     MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                           MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                           int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm PMPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int PMPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int PMPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int PMPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int PMPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int PMPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int PMPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                   int root, MPI_Comm comm, MPI_Comm *intercomm,
                                   int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                            const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                            int root, MPI_Comm comm, MPI_Comm *intercomm,
                                            int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int PMPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                         void *result_addr, MPI_Datatype datatype, int target_rank,
                                         MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int PMPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler PMPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int PMPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int PMPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                     int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File PMPI_File_f2c(int file);
__attribute__((visibility("default"))) int PMPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int PMPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                  MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int PMPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int PMPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                      MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                      MPI_Datatype *etype,
                                      MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int PMPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                          int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                       int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                          int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                           int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all(MPI_File fh, void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all(MPI_File fh, const void *buf, int count,
                                       MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread(MPI_File fh, void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                    MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_all(MPI_File fh, void *buf, int count,
                                       MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                        MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                             MPI_Offset *disp);
__attribute__((visibility("default"))) int PMPI_File_read_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                                int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                             MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                 MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                             MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int PMPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int PMPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int PMPI_Finalize(void);
__attribute__((visibility("default"))) int PMPI_Finalized(int *flag);
__attribute__((visibility("default"))) int PMPI_Free_mem(void *base);
__attribute__((visibility("default"))) int PMPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int PMPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype,
                                     int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype,
                                       MPI_Count *count);
__attribute__((visibility("default"))) int PMPI_Get(void *origin_addr, int origin_count,
                            MPI_Datatype origin_datatype, int target_rank,
                            MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int PMPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                     const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int PMPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                  int index[], int edges[]);
__attribute__((visibility("default"))) int PMPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                  int *newrank);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                        int neighbors[]);
__attribute__((visibility("default"))) int PMPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int PMPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int PMPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                       MPI_Grequest_free_function *free_fn,
                                       MPI_Grequest_cancel_function *cancel_fn,
                                       void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int PMPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int PMPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group PMPI_Group_f2c(int group);
__attribute__((visibility("default"))) int PMPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                           MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int PMPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int PMPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                              MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int PMPI_Group_union(MPI_Group group1, MPI_Group group2,
                                    MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Improbe(int source, int tag, MPI_Comm comm,
                                int *flag, MPI_Message *message,
                                MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Imrecv(void *buf, int count, MPI_Datatype type,
                               MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int PMPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info PMPI_Info_f2c(int info);
__attribute__((visibility("default"))) int PMPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                 char *value, int *flag);
__attribute__((visibility("default"))) int PMPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int PMPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int PMPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                          int *flag);
__attribute__((visibility("default"))) int PMPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int PMPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int PMPI_Initialized(int *flag);
__attribute__((visibility("default"))) int PMPI_Init_thread(int *argc, char ***argv, int required,
                                    int *provided);
__attribute__((visibility("default"))) int PMPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                         MPI_Comm bridge_comm, int remote_leader,
                                         int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                        MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int PMPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int PMPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message PMPI_Message_f2c(int message);
__attribute__((visibility("default"))) int PMPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Mrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                             void *recvbuf, const int recvcounts[], const int displs[],
                                             MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                            void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int PMPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int PMPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op PMPI_Op_f2c(int op);
__attribute__((visibility("default"))) int PMPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                      MPI_Datatype datatype, void *outbuf,
                                      MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int PMPI_Pack_external_size(const char datarep[], int incount,
                                           MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int PMPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                             void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                  int *size);
__attribute__((visibility("default"))) int PMPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int PMPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Publish_name(const char *service_name, MPI_Info info,
                                     const char *port_name);
__attribute__((visibility("default"))) int PMPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int PMPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                    int target_rank, MPI_Aint target_disp, int target_count,
                                    MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                  int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                     MPI_Datatype datatype, MPI_Op);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                       MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                        MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Register_datarep(const char *datarep,
                                         MPI_Datarep_conversion_function *read_conversion_fn,
                                         MPI_Datarep_conversion_function *write_conversion_fn,
                                         MPI_Datarep_extent_function *dtype_file_extent_fn,
                                         void *extra_state);
__attribute__((visibility("default"))) int PMPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request PMPI_Request_f2c(int request);
__attribute__((visibility("default"))) int PMPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Request_get_status(MPI_Request request, int *flag,
                                           MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                             MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                        void *result_addr, int result_count, MPI_Datatype result_datatype,
                                        int target_rank, MPI_Aint target_disp, int target_count,
                                        MPI_Datatype target_datatype, MPI_Op op,
                                        MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_cout,
                             MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 int dest, int sendtag, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int source, int recvtag,
                                 MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                         int dest, int sendtag, int source, int recvtag,
                                         MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int PMPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int PMPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int PMPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int PMPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                            int count);
__attribute__((visibility("default"))) int PMPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                              MPI_Count count);
__attribute__((visibility("default"))) int PMPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Testany(int count, MPI_Request array_of_requests[], int *index, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int PMPI_Testsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int PMPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                        MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_darray(int size, int rank, int ndims,
                                           const int gsize_array[], const int distrib_array[],
                                           const int darg_array[], const int psize_array[],
                                           int order, MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                             const MPI_Aint array_of_displacements[],
                                             MPI_Datatype oldtype,
                                             MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                           MPI_Type_delete_attr_function *type_delete_attr_fn,
                                           int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed_block(int count, int blocklength,
                                                   const MPI_Aint array_of_displacements[],
                                                   MPI_Datatype oldtype,
                                                   MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_indexed_block(int count, int blocklength,
                                                  const int array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                           const MPI_Aint array_of_displacements[],
                                           const MPI_Datatype array_of_types[],
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                             const int start_array[], int order,
                                             MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                            MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int PMPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype PMPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int PMPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                          int max_addresses, int max_datatypes,
                                          int array_of_integers[],
                                          MPI_Aint array_of_addresses[],
                                          MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int PMPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                          int *num_addresses, int *num_datatypes,
                                          int *combiner);
__attribute__((visibility("default"))) int PMPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                        MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                          MPI_Count *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_name(MPI_Datatype type, char *type_name,
                                      int *resultlen);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                             MPI_Aint *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                               MPI_Count *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_indexed(int count, const int array_of_blocklengths[],
                                     const int array_of_displacements[],
                                     MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                      void *attr_val);
__attribute__((visibility("default"))) int PMPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int PMPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int PMPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int PMPI_Type_vector(int count, int blocklength, int stride,
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Unpack(const void *inbuf, int insize, int *position,
                               void *outbuf, int outcount, MPI_Datatype datatype,
                               MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Unpublish_name(const char *service_name, MPI_Info info,
                                       const char *port_name);
__attribute__((visibility("default"))) int PMPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                         MPI_Aint *position, void *outbuf, int outcount,
                                         MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Waitall(int count, MPI_Request array_of_requests[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Waitany(int count, MPI_Request array_of_requests[],
                                int *index, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                     MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                            MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int PMPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int PMPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                   MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                          MPI_Win_delete_attr_function *win_delete_attr_fn,
                                          int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win PMPI_Win_f2c(int win);
__attribute__((visibility("default"))) int PMPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_get_attr(MPI_Win win, int win_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int PMPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int PMPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double PMPI_Wtick(void);
__attribute__((visibility("default"))) double PMPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int PMPI_T_finalize (void);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                         int *verbosity, MPI_Datatype *datatype,
                                         MPI_T_enum *enumtype, char *desc,
                                         int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int PMPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int PMPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int PMPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                            char *desc, int *desc_len, int *num_cvars,
                                            int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int PMPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int PMPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                        int *verbosity, int *var_class, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                        int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                            void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                     const void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                         void *buf);
__attribute__((visibility("default"))) int PMPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int PMPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                        int *name_len);
__attribute__((visibility("default"))) int MPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int MPI_T_finalize (void);
__attribute__((visibility("default"))) int MPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int MPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                        int *verbosity, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc,
                                        int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int MPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int MPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int MPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int MPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                           char *desc, int *desc_len, int *num_cvars,
                                           int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int MPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int MPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int MPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int MPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                       int *verbosity, int *var_class, MPI_Datatype *datatype,
                                       MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                       int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int MPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int MPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                           void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                   void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    const void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                        void *buf);
__attribute__((visibility("default"))) int MPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int MPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                       int *name_len);
__attribute__((visibility("default"))) int MPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("MPI_Attr_delete was deprecated in MPI-2.0; use MPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("PMPI_Attr_delete was deprecated in MPI-2.0; use PMPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("MPI_Attr_get was deprecated in MPI-2.0; use MPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("PMPI_Attr_get was deprecated in MPI-2.0; use PMPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("MPI_Attr_put was deprecated in MPI-2.0; use MPI_Comm_set_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("PMPI_Attr_put was deprecated in MPI-2.0; use PMPI_Comm_set_attr instead")));
typedef int (MPI_Copy_function)(MPI_Comm, int, void *,
                                void *, void *, int *);
typedef int (MPI_Delete_function)(MPI_Comm, int, void *, void *);
__attribute__((visibility("default"))) int MPI_Keyval_create(MPI_Copy_function *copy_fn,
                                     MPI_Delete_function *delete_fn,
                                     int *keyval, void *extra_state)
            __attribute__((__deprecated__("MPI_Keyval_create was deprecated in MPI-2.0; use MPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_create(MPI_Copy_function *copy_fn,
                                      MPI_Delete_function *delete_fn,
                                      int *keyval, void *extra_state)
            __attribute__((__deprecated__("PMPI_Keyval_create was deprecated in MPI-2.0; use PMPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int MPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("MPI_Keyval_free was deprecated in MPI-2.0; MPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("PMPI_Keyval_free was deprecated in MPI-2.0; PMPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_DUP_FN( MPI_Comm comm, int comm_keyval,
                                     void* extra_state,
                                     void* attribute_val_in,
                                     void* attribute_val_out,
                                     int* flag )
            __attribute__((__deprecated__("MPI_DUP_FN was deprecated in MPI-2.0; use MPI_COMM_DUP_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_COPY_FN( MPI_Comm comm, int comm_keyval,
                                           void* extra_state,
                                           void* attribute_val_in,
                                           void* attribute_val_out,
                                           int* flag )
            __attribute__((__deprecated__("MPI_NULL_COPY_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_COPY_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_DELETE_FN( MPI_Comm comm, int comm_keyval,
                                             void* attribute_val_out,
                                             void* extra_state )
            __attribute__((__deprecated__("MPI_NULL_DELETE_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_DELETE_FN instead.")));
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['mpi.h'] in ['/usr/include', '/usr/lib/openmpi']
                  Popping language C
================================================================================
TEST configureConversion from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:252)
TESTING: configureConversion from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:252)
  Check for the functions which convert communicators between C and Fortran
       - Define HAVE_MPI_COMM_F2C and HAVE_MPI_COMM_C2F if they are present
       - Some older MPI 1 implementations are missing these
                              All intermediate test results are stored in /tmp/petsc-VuReVk/config.packages.MPI
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Comm_f2c((MPI_Fint)0));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_COMM_F2C" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Comm_c2f(MPI_COMM_WORLD));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_COMM_C2F" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_Fint a;
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_FINT" to "1"
================================================================================
TEST checkMPICHorOpenMPI from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:461)
TESTING: checkMPICHorOpenMPI from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:461)
  Determine if MPICH_NUMVERSION or OMPI_MAJOR_VERSION exist in mpi.h
       Used for consistency checking of MPI installation at compile time
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:4:17: error: ???I_MPI_NUMVERSION??? undeclared here (not in a function); did you mean ???MPI_SUBVERSION????
 int mpich_ver = I_MPI_NUMVERSION;
                 ^~~~~~~~~~~~~~~~
                 MPI_SUBVERSION
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>
int mpich_ver = I_MPI_NUMVERSION;

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:4:17: error: ???MVAPICH2_NUMVERSION??? undeclared here (not in a function); did you mean ???MPI_SUBVERSION????
 int mpich_ver = MVAPICH2_NUMVERSION;
                 ^~~~~~~~~~~~~~~~~~~
                 MPI_SUBVERSION
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>
int mpich_ver = MVAPICH2_NUMVERSION;

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler: exit code 256
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:4:17: error: ???MPICH_NUMVERSION??? undeclared here (not in a function); did you mean ???MPI_SUBVERSION????
 int mpich_ver = MPICH_NUMVERSION;
                 ^~~~~~~~~~~~~~~~
                 MPI_SUBVERSION
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>
int mpich_ver = MPICH_NUMVERSION;

int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>
int ompi_major = OMPI_MAJOR_VERSION;
int ompi_minor = OMPI_MINOR_VERSION;
int ompi_release = OMPI_RELEASE_VERSION;

int main() {
;
  return 0;
}
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>
int ompi_major = OMPI_MAJOR_VERSION;
int ompi_minor = OMPI_MINOR_VERSION;
int ompi_release = OMPI_RELEASE_VERSION;
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.headers   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.packages.MPI/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.packages.MPI/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 2
# 1 "/usr/include/mpi.h" 1 3 4
# 232 "/usr/include/mpi.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long int ptrdiff_t;
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef struct {
  long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
  long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
# 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
} max_align_t;
# 233 "/usr/include/mpi.h" 2 3 4
# 265 "/usr/include/mpi.h" 3 4
# 1 "/usr/include/mpi_portable_platform.h" 1 3 4
# 266 "/usr/include/mpi.h" 2 3 4
# 342 "/usr/include/mpi.h" 3 4
typedef ptrdiff_t MPI_Aint;
typedef long long MPI_Offset;
typedef long long MPI_Count;
typedef struct ompi_communicator_t *MPI_Comm;
typedef struct ompi_datatype_t *MPI_Datatype;
typedef struct ompi_errhandler_t *MPI_Errhandler;
typedef struct ompi_file_t *MPI_File;
typedef struct ompi_group_t *MPI_Group;
typedef struct ompi_info_t *MPI_Info;
typedef struct ompi_op_t *MPI_Op;
typedef struct ompi_request_t *MPI_Request;
typedef struct ompi_message_t *MPI_Message;
typedef struct ompi_status_public_t MPI_Status;
typedef struct ompi_win_t *MPI_Win;
typedef struct mca_base_var_enum_t *MPI_T_enum;
typedef struct ompi_mpit_cvar_handle_t *MPI_T_cvar_handle;
typedef struct mca_base_pvar_handle_t *MPI_T_pvar_handle;
typedef struct mca_base_pvar_session_t *MPI_T_pvar_session;
struct ompi_status_public_t {
    int MPI_SOURCE;
    int MPI_TAG;
    int MPI_ERROR;
    int _cancelled;
    size_t _ucount;
};
typedef struct ompi_status_public_t ompi_status_public_t;
typedef int (MPI_Datarep_extent_function)(MPI_Datatype, MPI_Aint *, void *);
typedef int (MPI_Datarep_conversion_function)(void *, MPI_Datatype,
                                              int, void *, MPI_Offset, void *);
typedef void (MPI_Comm_errhandler_function)(MPI_Comm *, int *, ...);
typedef void (ompi_file_errhandler_fn)(MPI_File *, int *, ...);
typedef void (MPI_Win_errhandler_function)(MPI_Win *, int *, ...);
typedef void (MPI_User_function)(void *, void *, int *, MPI_Datatype *);
typedef int (MPI_Comm_copy_attr_function)(MPI_Comm, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Comm_delete_attr_function)(MPI_Comm, int, void *, void *);
typedef int (MPI_Type_copy_attr_function)(MPI_Datatype, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Type_delete_attr_function)(MPI_Datatype, int,
                                              void *, void *);
typedef int (MPI_Win_copy_attr_function)(MPI_Win, int, void *,
                                           void *, void *, int *);
typedef int (MPI_Win_delete_attr_function)(MPI_Win, int, void *, void *);
typedef int (MPI_Grequest_query_function)(void *, MPI_Status *);
typedef int (MPI_Grequest_free_function)(void *);
typedef int (MPI_Grequest_cancel_function)(void *, int);
typedef MPI_Comm_errhandler_function MPI_Comm_errhandler_fn
        __attribute__((__deprecated__("MPI_Comm_errhandler_fn was deprecated in MPI-2.2; use MPI_Comm_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_fn
        __attribute__((__deprecated__("MPI_File_errhandler_fn was deprecated in MPI-2.2; use MPI_File_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_function;
typedef MPI_Win_errhandler_function MPI_Win_errhandler_fn
        __attribute__((__deprecated__("MPI_Win_errhandler_fn was deprecated in MPI-2.2; use MPI_Win_errhandler_function instead")));
# 509 "/usr/include/mpi.h" 3 4
enum {
    MPI_TAG_UB,
    MPI_HOST,
    MPI_IO,
    MPI_WTIME_IS_GLOBAL,
    MPI_APPNUM,
    MPI_LASTUSEDCODE,
    MPI_UNIVERSE_SIZE,
    MPI_WIN_BASE,
    MPI_WIN_SIZE,
    MPI_WIN_DISP_UNIT,
    MPI_WIN_CREATE_FLAVOR,
    MPI_WIN_MODEL,
    IMPI_CLIENT_SIZE,
    IMPI_CLIENT_COLOR,
    IMPI_HOST_SIZE,
    IMPI_HOST_COLOR
};
# 623 "/usr/include/mpi.h" 3 4
enum {
  MPI_IDENT,
  MPI_CONGRUENT,
  MPI_SIMILAR,
  MPI_UNEQUAL
};
enum {
  MPI_THREAD_SINGLE,
  MPI_THREAD_FUNNELED,
  MPI_THREAD_SERIALIZED,
  MPI_THREAD_MULTIPLE
};
enum {
  MPI_COMBINER_NAMED,
  MPI_COMBINER_DUP,
  MPI_COMBINER_CONTIGUOUS,
  MPI_COMBINER_VECTOR,
  MPI_COMBINER_HVECTOR_INTEGER,
  MPI_COMBINER_HVECTOR,
  MPI_COMBINER_INDEXED,
  MPI_COMBINER_HINDEXED_INTEGER,
  MPI_COMBINER_HINDEXED,
  MPI_COMBINER_INDEXED_BLOCK,
  MPI_COMBINER_STRUCT_INTEGER,
  MPI_COMBINER_STRUCT,
  MPI_COMBINER_SUBARRAY,
  MPI_COMBINER_DARRAY,
  MPI_COMBINER_F90_REAL,
  MPI_COMBINER_F90_COMPLEX,
  MPI_COMBINER_F90_INTEGER,
  MPI_COMBINER_RESIZED,
  MPI_COMBINER_HINDEXED_BLOCK
};
enum {
  MPI_COMM_TYPE_SHARED,
  OMPI_COMM_TYPE_HWTHREAD,
  OMPI_COMM_TYPE_CORE,
  OMPI_COMM_TYPE_L1CACHE,
  OMPI_COMM_TYPE_L2CACHE,
  OMPI_COMM_TYPE_L3CACHE,
  OMPI_COMM_TYPE_SOCKET,
  OMPI_COMM_TYPE_NUMA,
  OMPI_COMM_TYPE_BOARD,
  OMPI_COMM_TYPE_HOST,
  OMPI_COMM_TYPE_CU,
  OMPI_COMM_TYPE_CLUSTER
};
enum {
  MPI_T_VERBOSITY_USER_BASIC,
  MPI_T_VERBOSITY_USER_DETAIL,
  MPI_T_VERBOSITY_USER_ALL,
  MPI_T_VERBOSITY_TUNER_BASIC,
  MPI_T_VERBOSITY_TUNER_DETAIL,
  MPI_T_VERBOSITY_TUNER_ALL,
  MPI_T_VERBOSITY_MPIDEV_BASIC,
  MPI_T_VERBOSITY_MPIDEV_DETAIL,
  MPI_T_VERBOSITY_MPIDEV_ALL
};
enum {
  MPI_T_SCOPE_CONSTANT,
  MPI_T_SCOPE_READONLY,
  MPI_T_SCOPE_LOCAL,
  MPI_T_SCOPE_GROUP,
  MPI_T_SCOPE_GROUP_EQ,
  MPI_T_SCOPE_ALL,
  MPI_T_SCOPE_ALL_EQ
};
enum {
  MPI_T_BIND_NO_OBJECT,
  MPI_T_BIND_MPI_COMM,
  MPI_T_BIND_MPI_DATATYPE,
  MPI_T_BIND_MPI_ERRHANDLER,
  MPI_T_BIND_MPI_FILE,
  MPI_T_BIND_MPI_GROUP,
  MPI_T_BIND_MPI_OP,
  MPI_T_BIND_MPI_REQUEST,
  MPI_T_BIND_MPI_WIN,
  MPI_T_BIND_MPI_MESSAGE,
  MPI_T_BIND_MPI_INFO
};
enum {
  MPI_T_PVAR_CLASS_STATE,
  MPI_T_PVAR_CLASS_LEVEL,
  MPI_T_PVAR_CLASS_SIZE,
  MPI_T_PVAR_CLASS_PERCENTAGE,
  MPI_T_PVAR_CLASS_HIGHWATERMARK,
  MPI_T_PVAR_CLASS_LOWWATERMARK,
  MPI_T_PVAR_CLASS_COUNTER,
  MPI_T_PVAR_CLASS_AGGREGATE,
  MPI_T_PVAR_CLASS_TIMER,
  MPI_T_PVAR_CLASS_GENERIC
};
# 820 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_DELETE_FN( MPI_Datatype datatype,
                                                  int type_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_COPY_FN( MPI_Datatype datatype,
                                                int type_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_DUP_FN( MPI_Datatype datatype,
                                          int type_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_DELETE_FN( MPI_Comm comm,
                                                  int comm_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_COPY_FN( MPI_Comm comm,
                                                int comm_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_DUP_FN( MPI_Comm comm, int comm_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_DELETE_FN( MPI_Win window,
                                                 int win_keyval,
                                                 void* attribute_val_out,
                                                 void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_COPY_FN( MPI_Win window, int win_keyval,
                                               void* extra_state,
                                               void* attribute_val_in,
                                               void* attribute_val_out,
                                               int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_DUP_FN( MPI_Win window, int win_keyval,
                                         void* extra_state,
                                         void* attribute_val_in,
                                         void* attribute_val_out,
                                         int* flag );
# 874 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_world;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_self;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_empty;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_request_t ompi_request_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_no_proc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_min;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_max;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_sum;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_prod;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_land;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_band;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_maxloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_minloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_replace;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_no_op;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_datatype_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_signed_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_byte;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_wchar;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_packed;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_character;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_longdbl_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex32;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_aint;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_offset;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_count;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_float_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_long_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errhandler_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_are_fatal;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_return;
__attribute__((visibility("default"))) extern struct ompi_predefined_win_t ompi_mpi_win_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_file_t ompi_mpi_file_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_env;
__attribute__((visibility("default"))) extern int *MPI_F_STATUS_IGNORE;
__attribute__((visibility("default"))) extern int *MPI_F_STATUSES_IGNORE;
# 1185 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int MPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                  int target_rank, MPI_Aint target_disp, int target_count,
                                  MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int MPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int MPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int MPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                 void *baseptr);
__attribute__((visibility("default"))) int MPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                             int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                             int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int MPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int MPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                   const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int MPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                int periods[], int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                const int periods[], int *newrank);
__attribute__((visibility("default"))) int MPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int MPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                  int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int MPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int MPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int MPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int MPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                   MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int MPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                          MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                          int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm MPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int MPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                         const int degrees[], const int targets[],
                                         const int weights[], MPI_Info info,
                                         int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int MPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                  int indegree, const int sources[],
                                                  const int sourceweights[],
                                                  int outdegree,
                                                  const int destinations[],
                                                  const int destweights[],
                                                  MPI_Info info, int reorder,
                                                  MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                           int sources[], int sourceweights[],
                                           int maxoutdegree,
                                           int destinations[],
                                           int destweights[]);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                  int *inneighbors,
                                                  int *outneighbors,
                                                  int *weighted);
__attribute__((visibility("default"))) int MPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int MPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int MPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int MPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int MPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int MPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                  int root, MPI_Comm comm, MPI_Comm *intercomm,
                                  int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                           const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                           int root, MPI_Comm comm, MPI_Comm *intercomm,
                                           int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int MPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                        void *result_addr, MPI_Datatype datatype, int target_rank,
                                        MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int MPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler MPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int MPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int MPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int MPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                    int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File MPI_File_f2c(int file);
__attribute__((visibility("default"))) int MPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int MPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                 MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int MPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int MPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                     MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                     MPI_Datatype *etype,
                                     MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int MPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                    int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                        int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read(MPI_File fh, void *buf, int count,
                                 MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all(MPI_File fh, void *buf, int count,
                                     MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write(MPI_File fh, const void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all(MPI_File fh, const void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_all(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                            MPI_Offset *disp);
__attribute__((visibility("default"))) int MPI_File_read_shared(MPI_File fh, void *buf, int count,
                                        MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_shared(MPI_File fh, const void *buf, int count,
      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                              int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                           MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                               MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                            MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int MPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int MPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int MPI_Finalize(void);
__attribute__((visibility("default"))) int MPI_Finalized(int *flag);
__attribute__((visibility("default"))) int MPI_Free_mem(void *base);
__attribute__((visibility("default"))) int MPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int MPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype, MPI_Count *count);
__attribute__((visibility("default"))) int MPI_Get(void *origin_addr, int origin_count,
                           MPI_Datatype origin_datatype, int target_rank,
                           MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                      void *result_addr, int result_count, MPI_Datatype result_datatype,
                                      int target_rank, MPI_Aint target_disp, int target_count,
                                      MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int MPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                    const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int MPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                 int index[], int edges[]);
__attribute__((visibility("default"))) int MPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                 int *newrank);
__attribute__((visibility("default"))) int MPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int MPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                       int neighbors[]);
__attribute__((visibility("default"))) int MPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int MPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int MPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                      MPI_Grequest_free_function *free_fn,
                                      MPI_Grequest_cancel_function *cancel_fn,
                                      void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int MPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int MPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group MPI_Group_f2c(int group);
__attribute__((visibility("default"))) int MPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                          MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int MPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int MPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                             MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int MPI_Group_union(MPI_Group group1, MPI_Group group2,
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Improbe(int source, int tag, MPI_Comm comm,
                               int *flag, MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Imrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int MPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int MPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info MPI_Info_f2c(int info);
__attribute__((visibility("default"))) int MPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                char *value, int *flag);
__attribute__((visibility("default"))) int MPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int MPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int MPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                         int *flag);
__attribute__((visibility("default"))) int MPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int MPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int MPI_Initialized(int *flag);
__attribute__((visibility("default"))) int MPI_Init_thread(int *argc, char ***argv, int required,
                                   int *provided);
__attribute__((visibility("default"))) int MPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                        MPI_Comm bridge_comm, int remote_leader,
                                        int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                       MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                              MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int MPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int MPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message MPI_Message_f2c(int message);
__attribute__((visibility("default"))) int MPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Mrecv(void *buf, int count, MPI_Datatype type,
                             MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int displs[],
                                           MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                         void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                         MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                          void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                          void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int MPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int MPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op MPI_Op_f2c(int op);
__attribute__((visibility("default"))) int MPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                     MPI_Datatype datatype, void *outbuf,
                                     MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int MPI_Pack_external_size(const char datarep[], int incount,
                                          MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int MPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                            void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                 int *size);
__attribute__((visibility("default"))) int MPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Publish_name(const char *service_name, MPI_Info info,
                                    const char *port_name);
__attribute__((visibility("default"))) int MPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                           int target_rank, MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int MPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                 int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                            int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                    MPI_Datatype datatype, MPI_Op op);
__attribute__((visibility("default"))) int MPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Register_datarep(const char *datarep,
                                        MPI_Datarep_conversion_function *read_conversion_fn,
                                        MPI_Datarep_conversion_function *write_conversion_fn,
                                        MPI_Datarep_extent_function *dtype_file_extent_fn,
                                        void *extra_state);
__attribute__((visibility("default"))) int MPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request MPI_Request_f2c(int request);
__attribute__((visibility("default"))) int MPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Request_get_status(MPI_Request request, int *flag,
                                          MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                            MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op,
                                       MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_cout,
                            MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                 int dest, int tag, MPI_Comm comm,
                                 MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                            int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                int dest, int sendtag, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int source, int recvtag,
                                MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                        int dest, int sendtag, int source, int recvtag,
                                        MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int MPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int MPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int MPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int MPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                           int count);
__attribute__((visibility("default"))) int MPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                             MPI_Count count);
__attribute__((visibility("default"))) int MPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                               MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Testany(int count, MPI_Request array_of_requests[], int *index,
                               int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int MPI_Testsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int MPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                       MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_darray(int size, int rank, int ndims,
                                          const int gsize_array[], const int distrib_array[],
                                          const int darg_array[], const int psize_array[],
                                          int order, MPI_Datatype oldtype,
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed_block(int count, int blocklength,
                                                  const MPI_Aint array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                            const MPI_Aint array_of_displacements[],
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                           MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                          MPI_Type_delete_attr_function *type_delete_attr_fn,
                                          int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Type_create_indexed_block(int count, int blocklength,
                                                 const int array_of_displacements[],
                                                 MPI_Datatype oldtype,
                                                 MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                          const MPI_Aint array_of_displacements[],
                                          const MPI_Datatype array_of_types[],
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                            const int start_array[], int order,
                                            MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                           MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int MPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype MPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int MPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                         int max_addresses, int max_datatypes,
                                         int array_of_integers[],
                                         MPI_Aint array_of_addresses[],
                                         MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int MPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                         int *num_addresses, int *num_datatypes,
                                         int *combiner);
__attribute__((visibility("default"))) int MPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                       MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                         MPI_Count *extent);
__attribute__((visibility("default"))) int MPI_Type_get_name(MPI_Datatype type, char *type_name,
                                     int *resultlen);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                            MPI_Aint *true_extent);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                              MPI_Count *true_extent);
__attribute__((visibility("default"))) int MPI_Type_indexed(int count, const int array_of_blocklengths[],
                                    const int array_of_displacements[],
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                     void *attr_val);
__attribute__((visibility("default"))) int MPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int MPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int MPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int MPI_Type_vector(int count, int blocklength, int stride,
                                   MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Unpack(const void *inbuf, int insize, int *position,
                              void *outbuf, int outcount, MPI_Datatype datatype,
                              MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Unpublish_name(const char *service_name, MPI_Info info, const char *port_name);
__attribute__((visibility("default"))) int MPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                        MPI_Aint *position, void *outbuf, int outcount,
                                        MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Waitall(int count, MPI_Request array_of_requests[],
                               MPI_Status *array_of_statuses);
__attribute__((visibility("default"))) int MPI_Waitany(int count, MPI_Request array_of_requests[],
                               int *index, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                    MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                           MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int MPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int MPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                  MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                             MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                         MPI_Win_delete_attr_function *win_delete_attr_fn,
                                         int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int MPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win MPI_Win_f2c(int win);
__attribute__((visibility("default"))) int MPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int MPI_Win_get_attr(MPI_Win win, int win_keyval,
                                    void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int MPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int MPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int MPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double MPI_Wtick(void);
__attribute__((visibility("default"))) double MPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int PMPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int PMPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int PMPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                  void *baseptr);
__attribute__((visibility("default"))) int PMPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                          const int degrees[], const int targets[],
                                          const int weights[], MPI_Info info,
                                          int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                   int indegree, const int sources[],
                                                   const int sourceweights[],
                                                   int outdegree,
                                                   const int destinations[],
                                                   const int destweights[],
                                                   MPI_Info info, int reorder,
                                                   MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                            int sources[], int sourceweights[],
                                            int maxoutdegree,
                                             int destinations[],
                                            int destweights[]);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                   int *inneighbors,
                                                   int *outneighbors,
                                                   int *weighted);
__attribute__((visibility("default"))) int PMPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                              int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int PMPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int PMPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                    const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int PMPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                 int periods[], int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                 const int periods[], int *newrank);
__attribute__((visibility("default"))) int PMPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int PMPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                   int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int PMPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int PMPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int PMPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int PMPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int PMPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                     MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                           MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                           int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm PMPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int PMPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int PMPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int PMPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int PMPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int PMPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int PMPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                   int root, MPI_Comm comm, MPI_Comm *intercomm,
                                   int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                            const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                            int root, MPI_Comm comm, MPI_Comm *intercomm,
                                            int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int PMPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                         void *result_addr, MPI_Datatype datatype, int target_rank,
                                         MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int PMPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler PMPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int PMPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int PMPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                     int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File PMPI_File_f2c(int file);
__attribute__((visibility("default"))) int PMPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int PMPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                  MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int PMPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int PMPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                      MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                      MPI_Datatype *etype,
                                      MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int PMPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                          int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                       int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                          int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                           int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all(MPI_File fh, void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all(MPI_File fh, const void *buf, int count,
                                       MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread(MPI_File fh, void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                    MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_all(MPI_File fh, void *buf, int count,
                                       MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                        MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                             MPI_Offset *disp);
__attribute__((visibility("default"))) int PMPI_File_read_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                                int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                             MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                 MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                             MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int PMPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int PMPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int PMPI_Finalize(void);
__attribute__((visibility("default"))) int PMPI_Finalized(int *flag);
__attribute__((visibility("default"))) int PMPI_Free_mem(void *base);
__attribute__((visibility("default"))) int PMPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int PMPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype,
                                     int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype,
                                       MPI_Count *count);
__attribute__((visibility("default"))) int PMPI_Get(void *origin_addr, int origin_count,
                            MPI_Datatype origin_datatype, int target_rank,
                            MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int PMPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                     const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int PMPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                  int index[], int edges[]);
__attribute__((visibility("default"))) int PMPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                  int *newrank);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                        int neighbors[]);
__attribute__((visibility("default"))) int PMPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int PMPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int PMPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                       MPI_Grequest_free_function *free_fn,
                                       MPI_Grequest_cancel_function *cancel_fn,
                                       void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int PMPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int PMPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group PMPI_Group_f2c(int group);
__attribute__((visibility("default"))) int PMPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                           MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int PMPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int PMPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                              MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int PMPI_Group_union(MPI_Group group1, MPI_Group group2,
                                    MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Improbe(int source, int tag, MPI_Comm comm,
                                int *flag, MPI_Message *message,
                                MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Imrecv(void *buf, int count, MPI_Datatype type,
                               MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int PMPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info PMPI_Info_f2c(int info);
__attribute__((visibility("default"))) int PMPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                 char *value, int *flag);
__attribute__((visibility("default"))) int PMPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int PMPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int PMPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                          int *flag);
__attribute__((visibility("default"))) int PMPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int PMPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int PMPI_Initialized(int *flag);
__attribute__((visibility("default"))) int PMPI_Init_thread(int *argc, char ***argv, int required,
                                    int *provided);
__attribute__((visibility("default"))) int PMPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                         MPI_Comm bridge_comm, int remote_leader,
                                         int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                        MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int PMPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int PMPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message PMPI_Message_f2c(int message);
__attribute__((visibility("default"))) int PMPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Mrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                             void *recvbuf, const int recvcounts[], const int displs[],
                                             MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                            void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int PMPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int PMPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op PMPI_Op_f2c(int op);
__attribute__((visibility("default"))) int PMPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                      MPI_Datatype datatype, void *outbuf,
                                      MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int PMPI_Pack_external_size(const char datarep[], int incount,
                                           MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int PMPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                             void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                  int *size);
__attribute__((visibility("default"))) int PMPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int PMPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Publish_name(const char *service_name, MPI_Info info,
                                     const char *port_name);
__attribute__((visibility("default"))) int PMPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int PMPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                    int target_rank, MPI_Aint target_disp, int target_count,
                                    MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                  int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                     MPI_Datatype datatype, MPI_Op);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                       MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                        MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Register_datarep(const char *datarep,
                                         MPI_Datarep_conversion_function *read_conversion_fn,
                                         MPI_Datarep_conversion_function *write_conversion_fn,
                                         MPI_Datarep_extent_function *dtype_file_extent_fn,
                                         void *extra_state);
__attribute__((visibility("default"))) int PMPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request PMPI_Request_f2c(int request);
__attribute__((visibility("default"))) int PMPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Request_get_status(MPI_Request request, int *flag,
                                           MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                             MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                        void *result_addr, int result_count, MPI_Datatype result_datatype,
                                        int target_rank, MPI_Aint target_disp, int target_count,
                                        MPI_Datatype target_datatype, MPI_Op op,
                                        MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_cout,
                             MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 int dest, int sendtag, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int source, int recvtag,
                                 MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                         int dest, int sendtag, int source, int recvtag,
                                         MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int PMPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int PMPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int PMPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int PMPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                            int count);
__attribute__((visibility("default"))) int PMPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                              MPI_Count count);
__attribute__((visibility("default"))) int PMPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Testany(int count, MPI_Request array_of_requests[], int *index, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int PMPI_Testsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int PMPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                        MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_darray(int size, int rank, int ndims,
                                           const int gsize_array[], const int distrib_array[],
                                           const int darg_array[], const int psize_array[],
                                           int order, MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                             const MPI_Aint array_of_displacements[],
                                             MPI_Datatype oldtype,
                                             MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                           MPI_Type_delete_attr_function *type_delete_attr_fn,
                                           int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed_block(int count, int blocklength,
                                                   const MPI_Aint array_of_displacements[],
                                                   MPI_Datatype oldtype,
                                                   MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_indexed_block(int count, int blocklength,
                                                  const int array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                           const MPI_Aint array_of_displacements[],
                                           const MPI_Datatype array_of_types[],
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                             const int start_array[], int order,
                                             MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                            MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int PMPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype PMPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int PMPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                          int max_addresses, int max_datatypes,
                                          int array_of_integers[],
                                          MPI_Aint array_of_addresses[],
                                          MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int PMPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                          int *num_addresses, int *num_datatypes,
                                          int *combiner);
__attribute__((visibility("default"))) int PMPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                        MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                          MPI_Count *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_name(MPI_Datatype type, char *type_name,
                                      int *resultlen);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                             MPI_Aint *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                               MPI_Count *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_indexed(int count, const int array_of_blocklengths[],
                                     const int array_of_displacements[],
                                     MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                      void *attr_val);
__attribute__((visibility("default"))) int PMPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int PMPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int PMPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int PMPI_Type_vector(int count, int blocklength, int stride,
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Unpack(const void *inbuf, int insize, int *position,
                               void *outbuf, int outcount, MPI_Datatype datatype,
                               MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Unpublish_name(const char *service_name, MPI_Info info,
                                       const char *port_name);
__attribute__((visibility("default"))) int PMPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                         MPI_Aint *position, void *outbuf, int outcount,
                                         MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Waitall(int count, MPI_Request array_of_requests[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Waitany(int count, MPI_Request array_of_requests[],
                                int *index, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                     MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                            MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int PMPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int PMPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                   MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                          MPI_Win_delete_attr_function *win_delete_attr_fn,
                                          int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win PMPI_Win_f2c(int win);
__attribute__((visibility("default"))) int PMPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_get_attr(MPI_Win win, int win_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int PMPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int PMPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double PMPI_Wtick(void);
__attribute__((visibility("default"))) double PMPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int PMPI_T_finalize (void);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                         int *verbosity, MPI_Datatype *datatype,
                                         MPI_T_enum *enumtype, char *desc,
                                         int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int PMPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int PMPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int PMPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                            char *desc, int *desc_len, int *num_cvars,
                                            int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int PMPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int PMPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                        int *verbosity, int *var_class, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                        int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                            void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                     const void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                         void *buf);
__attribute__((visibility("default"))) int PMPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int PMPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                        int *name_len);
__attribute__((visibility("default"))) int MPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int MPI_T_finalize (void);
__attribute__((visibility("default"))) int MPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int MPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                        int *verbosity, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc,
                                        int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int MPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int MPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int MPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int MPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                           char *desc, int *desc_len, int *num_cvars,
                                           int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int MPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int MPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int MPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int MPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                       int *verbosity, int *var_class, MPI_Datatype *datatype,
                                       MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                       int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int MPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int MPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                           void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                   void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    const void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                        void *buf);
__attribute__((visibility("default"))) int MPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int MPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                       int *name_len);
__attribute__((visibility("default"))) int MPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("MPI_Attr_delete was deprecated in MPI-2.0; use MPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("PMPI_Attr_delete was deprecated in MPI-2.0; use PMPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("MPI_Attr_get was deprecated in MPI-2.0; use MPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("PMPI_Attr_get was deprecated in MPI-2.0; use PMPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("MPI_Attr_put was deprecated in MPI-2.0; use MPI_Comm_set_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("PMPI_Attr_put was deprecated in MPI-2.0; use PMPI_Comm_set_attr instead")));
typedef int (MPI_Copy_function)(MPI_Comm, int, void *,
                                void *, void *, int *);
typedef int (MPI_Delete_function)(MPI_Comm, int, void *, void *);
__attribute__((visibility("default"))) int MPI_Keyval_create(MPI_Copy_function *copy_fn,
                                     MPI_Delete_function *delete_fn,
                                     int *keyval, void *extra_state)
            __attribute__((__deprecated__("MPI_Keyval_create was deprecated in MPI-2.0; use MPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_create(MPI_Copy_function *copy_fn,
                                      MPI_Delete_function *delete_fn,
                                      int *keyval, void *extra_state)
            __attribute__((__deprecated__("PMPI_Keyval_create was deprecated in MPI-2.0; use PMPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int MPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("MPI_Keyval_free was deprecated in MPI-2.0; MPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("PMPI_Keyval_free was deprecated in MPI-2.0; PMPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_DUP_FN( MPI_Comm comm, int comm_keyval,
                                     void* extra_state,
                                     void* attribute_val_in,
                                     void* attribute_val_out,
                                     int* flag )
            __attribute__((__deprecated__("MPI_DUP_FN was deprecated in MPI-2.0; use MPI_COMM_DUP_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_COPY_FN( MPI_Comm comm, int comm_keyval,
                                           void* extra_state,
                                           void* attribute_val_in,
                                           void* attribute_val_out,
                                           int* flag )
            __attribute__((__deprecated__("MPI_NULL_COPY_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_COPY_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_DELETE_FN( MPI_Comm comm, int comm_keyval,
                                             void* attribute_val_out,
                                             void* extra_state )
            __attribute__((__deprecated__("MPI_NULL_DELETE_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_DELETE_FN instead.")));
# 4 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 2
# 4 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
int ompi_major = 
# 4 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 3 4
                4
# 4 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
                                  ;
int ompi_minor = 
# 5 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 3 4
                0
# 5 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
                                  ;
int ompi_release = 
# 6 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c" 3 4
                  0
# 6 "/tmp/petsc-VuReVk/config.packages.MPI/conftest.c"
                                      ;
                  Defined "HAVE_OMPI_MAJOR_VERSION" to "4"
                  Defined "HAVE_OMPI_MINOR_VERSION" to "0"
                  Defined "HAVE_OMPI_RELEASE_VERSION" to "0"
================================================================================
TEST configureMPI2 from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:185)
TESTING: configureMPI2 from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:185)
  Check for functions added to the interface in MPI-2
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
int flag;if (MPI_Finalized(&flag));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_FINALIZED" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Allreduce(MPI_IN_PLACE,0, 1, MPI_INT, MPI_SUM, MPI_COMM_SELF));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_IN_PLACE" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:6:94: warning: initialization of ???struct ompi_datatype_t *??? from ???int??? makes pointer from integer without a cast [-Wint-conversion]
 int count=2; int blocklens[2]={0,1}; MPI_Aint indices[2]={0,1}; MPI_Datatype old_types[2]={0,1}; MPI_Datatype *newtype = 0;
                                                                                              ^
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:6:94: note: (near initialization for ???old_types[1]???)
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
int count=2; int blocklens[2]={0,1}; MPI_Aint indices[2]={0,1}; MPI_Datatype old_types[2]={0,1}; MPI_Datatype *newtype = 0;
                                              if (MPI_Type_create_struct(count, blocklens, indices, old_types, newtype));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:6:1: warning: ???MPI_Comm_errhandler_fn??? is deprecated: MPI_Comm_errhandler_fn was deprecated in MPI-2.2; use MPI_Comm_errhandler_function instead [-Wdeprecated-declarations]
 MPI_Comm_errhandler_fn * p_err_fun = 0; MPI_Errhandler * p_errhandler = 0; if (MPI_Comm_create_errhandler(p_err_fun,p_errhandler));
 ^~~~~~~~~~~~~~~~~~~~~~
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_Comm_errhandler_fn * p_err_fun = 0; MPI_Errhandler * p_errhandler = 0; if (MPI_Comm_create_errhandler(p_err_fun,p_errhandler));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Comm_set_errhandler(MPI_COMM_WORLD,MPI_ERRORS_RETURN));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Reduce_local(0, 0, 0, MPI_INT, MPI_SUM));;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_REDUCE_LOCAL" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.MPI -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
char version[MPI_MAX_LIBRARY_VERSION_STRING];int verlen;if (MPI_Get_library_version(version,&verlen));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_GET_LIBRARY_VERSION" to "1"
================================================================================
TEST configureMPI3 from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:224)
TESTING: configureMPI3 from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:224)
  Check for functions added to the interface in MPI-3
                  Checking for functions [MPI_Win_create] in library [] []
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_Win_create();
static void _check_MPI_Win_create() { MPI_Win_create(); }

int main() {
_check_MPI_Win_create();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lstdc++ -ldl 
                    Popping language C
                  Defined "HAVE_MPI_WIN_CREATE" to "1"
                  Defined "HAVE_MPI_REPLACE" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_Comm scomm; if (MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &scomm));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_SHARED_COMM" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_Win win; if (MPI_Win_allocate_shared(100,10,MPI_INFO_NULL,MPI_COMM_WORLD, 0, &win));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_WIN_ALLOCATE_SHARED" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
if (MPI_Win_shared_query(MPI_WIN_NULL,0,0,0,0));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPI_WIN_SHARED_QUERY" to "1"
                  Defined "HAVE_MPI_WIN_CREATE_FEATURE" to "1"
                  Defined "HAVE_MPI_PROCESS_SHARED_MEMORY" to "1"
================================================================================
TEST configureTypes from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:272)
TESTING: configureTypes from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:272)
  Checking for MPI types
Checking for size of type: MPI_Comm
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

#define MPICH_IGNORE_CXX_SEEK
#define MPICH_SKIP_MPICXX 1
#define OMPI_SKIP_MPICXX 1
#include <mpi.h>

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(MPI_Comm));
;
  return 0;
}
                                Pushing language C
                                Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                    Popping language C
                    Defined "SIZEOF_MPI_COMM" to "8"
Checking for size of type: MPI_Fint
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.types/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.types/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <sys/types.h>
#if STDC_HEADERS
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#endif

#define MPICH_IGNORE_CXX_SEEK
#define MPICH_SKIP_MPICXX 1
#define OMPI_SKIP_MPICXX 1
#include <mpi.h>

int main() {
FILE *f = fopen("conftestval", "w");

if (!f) exit(1);
fprintf(f, "%lu\n", (unsigned long)sizeof(MPI_Fint));
;
  return 0;
}
                                Pushing language C
                                Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.types/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.types/conftest.o -lstdc++ -ldl 
Testing executable /tmp/petsc-VuReVk/config.types/conftest to see if it can be run
Executing: /tmp/petsc-VuReVk/config.types/conftest
Executing: /tmp/petsc-VuReVk/config.types/conftest
                    Popping language C
                    Defined "SIZEOF_MPI_FINT" to "4"
================================================================================
TEST configureMPITypes from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:284)
TESTING: configureMPITypes from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:284)
  Checking for MPI Datatype handles
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_LONG_DOUBLE, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_LONG_DOUBLE, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_LONG_DOUBLE, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_LONG_DOUBLE, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.packages.MPI/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12: undefined reference to `MPI_Type_extent'
collect2: error: ld returned 1 exit status
                  Popping language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_INT64_T, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_INT64_T, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_INT64_T, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_INT64_T, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.packages.MPI/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12: undefined reference to `MPI_Type_extent'
collect2: error: ld returned 1 exit status
                  Popping language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_C_DOUBLE_COMPLEX, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_C_DOUBLE_COMPLEX, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12:8: warning: implicit declaration of function ???MPI_Type_extent???; did you mean ???MPI_Type_get_extent???? [-Wimplicit-function-declaration]
 ierr = MPI_Type_extent(MPI_C_DOUBLE_COMPLEX, &size);
        ^~~~~~~~~~~~~~~
        MPI_Type_get_extent
Source:
#include "confdefs.h"
#include "conffix.h"
#ifdef PETSC_HAVE_STDLIB_H
  #include <stdlib.h>
#endif
#include <mpi.h>

int main() {
MPI_Aint size;
int ierr;
MPI_Init(0,0);
ierr = MPI_Type_extent(MPI_C_DOUBLE_COMPLEX, &size);
if(ierr || (size == 0)) exit(1);
MPI_Finalize();
;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.packages.MPI/conftest.o: in function `main':
/tmp/petsc-VuReVk/config.packages.MPI/conftest.c:12: undefined reference to `MPI_Type_extent'
collect2: error: ld returned 1 exit status
                  Popping language C
================================================================================
TEST configureMissingPrototypes from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:352)
TESTING: configureMissingPrototypes from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:352)
  Checks for missing prototypes, which it adds to petscfix.h
================================================================================
TEST SGIMPICheck from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:371)
TESTING: SGIMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:371)
  Returns true if SGI MPI is used
                  Checking for functions [MPI_SGI_barrier] in library [] []
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_SGI_barrier();
static void _check_MPI_SGI_barrier() { MPI_SGI_barrier(); }

int main() {
_check_MPI_SGI_barrier();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_MPI_SGI_barrier':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `MPI_SGI_barrier'
collect2: error: ld returned 1 exit status
                    Popping language C
                SGI MPI test failure
================================================================================
TEST CxxMPICheck from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:381)
TESTING: CxxMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:381)
  Make sure C++ can compile and link
                  Pushing language Cxx
Checking for header mpi.h
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp -g -O        /tmp/petsc-VuReVk/config.libraries/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
;
  return 0;
}
Checking for C++ MPI_Finalize()
                  Checking for functions [MPI_Finalize] in library [] []
                    Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp -g -O        /tmp/petsc-VuReVk/config.libraries/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <mpi.h>
static void _check_MPI_Finalize() { int ierr;
ierr = MPI_Finalize();; }

int main() {
_check_MPI_Finalize();;
  return 0;
}
                            Pushing language CXX
                            Popping language CXX
Executing: mpicxx  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                    Popping language Cxx
                  Popping language Cxx
================================================================================
TEST FortranMPICheck from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:399)
TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:399)
  Make sure fortran include [mpif.h] and library symbols are found
                  Pushing language FC
Checking for header mpif.h
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp  -g -O    /tmp/petsc-VuReVk/config.libraries/conftest.F90 
Successful compile:
Source:
      program main
#include "mpif.h"
      end
Checking for fortran mpi_init()
                  Checking for functions [] in library [] []
                    Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp  -g -O    /tmp/petsc-VuReVk/config.libraries/conftest.F90 
Successful compile:
Source:
      program main
#include "mpif.h"
       integer ierr
       call mpi_init(ierr)
      end
                            Pushing language FC
                            Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                    Popping language FC
Checking for mpi.mod
                  Checking for functions [] in library [] []
                    Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp  -g -O    /tmp/petsc-VuReVk/config.libraries/conftest.F90 
Successful compile:
Source:
      program main
       use mpi
       integer ierr,rank
       call mpi_init(ierr)
       call mpi_comm_rank(MPI_COMM_WORLD,rank,ierr)

      end
                            Pushing language FC
                            Popping language FC
Executing: mpif90  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp  -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                    Popping language FC
                  Defined "HAVE_MPI_F90MODULE" to "1"
                  Popping language FC
================================================================================
TEST configureIO from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:424)
TESTING: configureIO from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:424)
  Check for the functions in MPI/IO
       - Define HAVE_MPIIO if they are present
       - Some older MPI 1 implementations are missing these
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_Aint lb, extent;
if (MPI_Type_get_extent(MPI_INT, &lb, &extent));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_File fh;
void *buf;
MPI_Status status;
if (MPI_File_write_all(fh, buf, 1, MPI_INT, &status));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_File fh;
void *buf;
MPI_Status status;
if (MPI_File_read_all(fh, buf, 1, MPI_INT, &status));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_File fh;
MPI_Offset disp;
MPI_Info info;
if (MPI_File_set_view(fh, disp, MPI_INT, MPI_INT, "", info));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_File fh;
MPI_Info info;
if (MPI_File_open(MPI_COMM_SELF, "", 0, info, &fh));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
MPI_File fh;
MPI_Info info;
if (MPI_File_close(&fh));
;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o  -lstdc++ -ldl 
                  Defined "HAVE_MPIIO" to "1"
================================================================================
TEST findMPIInc from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:498)
TESTING: findMPIInc from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:498)
  Find MPI include paths from "mpicc -show" and use with CUDAC_FLAGS
              Checking for functions [MPI_Alltoallw] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_Alltoallw();
static void _check_MPI_Alltoallw() { MPI_Alltoallw(); }

int main() {
_check_MPI_Alltoallw();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                Popping language C
              Checking for functions [MPI_Type_create_indexed_block] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_Type_create_indexed_block();
static void _check_MPI_Type_create_indexed_block() { MPI_Type_create_indexed_block(); }

int main() {
_check_MPI_Type_create_indexed_block();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                Popping language C
              Defined "HAVE_MPI_ALLTOALLW" to "1"
                    Checking for functions [MPI_Comm_spawn MPI_Type_get_envelope MPI_Type_get_extent MPI_Type_dup MPI_Init_thread MPI_Iallreduce MPI_Ibarrier MPI_Finalized MPI_Exscan MPI_Reduce_scatter MPI_Reduce_scatter_block] in library [] []
                      Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPI_Comm_spawn();
static void _check_MPI_Comm_spawn() { MPI_Comm_spawn(); }
char MPI_Type_get_envelope();
static void _check_MPI_Type_get_envelope() { MPI_Type_get_envelope(); }
char MPI_Type_get_extent();
static void _check_MPI_Type_get_extent() { MPI_Type_get_extent(); }
char MPI_Type_dup();
static void _check_MPI_Type_dup() { MPI_Type_dup(); }
char MPI_Init_thread();
static void _check_MPI_Init_thread() { MPI_Init_thread(); }
char MPI_Iallreduce();
static void _check_MPI_Iallreduce() { MPI_Iallreduce(); }
char MPI_Ibarrier();
static void _check_MPI_Ibarrier() { MPI_Ibarrier(); }
char MPI_Finalized();
static void _check_MPI_Finalized() { MPI_Finalized(); }
char MPI_Exscan();
static void _check_MPI_Exscan() { MPI_Exscan(); }
char MPI_Reduce_scatter();
static void _check_MPI_Reduce_scatter() { MPI_Reduce_scatter(); }
char MPI_Reduce_scatter_block();
static void _check_MPI_Reduce_scatter_block() { MPI_Reduce_scatter_block(); }

int main() {
_check_MPI_Comm_spawn();
_check_MPI_Type_get_envelope();
_check_MPI_Type_get_extent();
_check_MPI_Type_dup();
_check_MPI_Init_thread();
_check_MPI_Iallreduce();
_check_MPI_Ibarrier();
_check_MPI_Finalized();
_check_MPI_Exscan();
_check_MPI_Reduce_scatter();
_check_MPI_Reduce_scatter_block();;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                      Popping language C
              Defined "HAVE_MPI_COMM_SPAWN" to "1"
              Defined "HAVE_MPI_TYPE_GET_ENVELOPE" to "1"
              Defined "HAVE_MPI_TYPE_GET_EXTENT" to "1"
              Defined "HAVE_MPI_TYPE_DUP" to "1"
              Defined "HAVE_MPI_INIT_THREAD" to "1"
              Defined "HAVE_MPI_IALLREDUCE" to "1"
              Defined "HAVE_MPI_IBARRIER" to "1"
              Defined "HAVE_MPI_FINALIZED" to "1"
              Defined "HAVE_MPI_EXSCAN" to "1"
              Defined "HAVE_MPI_REDUCE_SCATTER" to "1"
              Defined "HAVE_MPI_REDUCE_SCATTER_BLOCK" to "1"
              Checking for functions [MPIX_Iallreduce] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPIX_Iallreduce();
static void _check_MPIX_Iallreduce() { MPIX_Iallreduce(); }

int main() {
_check_MPIX_Iallreduce();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_MPIX_Iallreduce':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `MPIX_Iallreduce'
collect2: error: ld returned 1 exit status
                Popping language C
              Checking for functions [MPIX_Ibarrier] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPIX_Ibarrier();
static void _check_MPIX_Ibarrier() { MPIX_Ibarrier(); }

int main() {
_check_MPIX_Ibarrier();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_MPIX_Ibarrier':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `MPIX_Ibarrier'
collect2: error: ld returned 1 exit status
                Popping language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
int combiner = MPI_COMBINER_DUP;;
  return 0;
}
              Defined "HAVE_MPI_COMBINER_DUP" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
int combiner = MPI_COMBINER_CONTIGUOUS;;
  return 0;
}
              Defined "HAVE_MPI_COMBINER_CONTIGUOUS" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

int main() {
int combiner = MPI_COMBINER_NAMED;;
  return 0;
}
              Defined "HAVE_MPI_COMBINER_NAMED" to "1"
              Checking for functions [MPIDI_CH3I_sock_set] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPIDI_CH3I_sock_set();
static void _check_MPIDI_CH3I_sock_set() { MPIDI_CH3I_sock_set(); }

int main() {
_check_MPIDI_CH3I_sock_set();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_MPIDI_CH3I_sock_set':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `MPIDI_CH3I_sock_set'
collect2: error: ld returned 1 exit status
                Popping language C
              Checking for functions [MPIDI_CH3I_sock_fixed_nbc_progress] in library [] []
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char MPIDI_CH3I_sock_fixed_nbc_progress();
static void _check_MPIDI_CH3I_sock_fixed_nbc_progress() { MPIDI_CH3I_sock_fixed_nbc_progress(); }

int main() {
_check_MPIDI_CH3I_sock_fixed_nbc_progress();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_MPIDI_CH3I_sock_fixed_nbc_progress':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `MPIDI_CH3I_sock_fixed_nbc_progress'
collect2: error: ld returned 1 exit status
                Popping language C
================================================================================
TEST checkSharedLibrary from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:133)
TESTING: checkSharedLibrary from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:133)
  Sets flag indicating if MPI libraries are shared or not and
    determines if MPI libraries CANNOT be used by shared libraries
================================================================================
TEST configureMPIEXEC from config.packages.MPI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/MPI.py:146)
TESTING: configureMPIEXEC from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:146)
  Checking for mpiexec
                  Pushing language C
                  Popping language C
Checking for program /home/ryan/bin/mpiexec...not found
Checking for program /home/ryan/bin/mpirun...not found
Checking for program /home/ryan/bin/mprun...not found
Checking for program /home/ryan/bin/mpiexec...not found
Checking for program /home/ryan/bin/mpirun...not found
Checking for program /home/ryan/bin/mprun...not found
Checking for program /usr/local/bin/mpiexec...not found
Checking for program /usr/local/bin/mpirun...not found
Checking for program /usr/local/bin/mprun...not found
Checking for program /usr/local/bin/mpiexec...not found
Checking for program /usr/local/bin/mpirun...not found
Checking for program /usr/local/bin/mprun...not found
Checking for program /home/ryan/bin/mpiexec...not found
Checking for program /home/ryan/bin/mpirun...not found
Checking for program /home/ryan/bin/mprun...not found
Checking for program /home/ryan/bin/mpiexec...not found
Checking for program /home/ryan/bin/mpirun...not found
Checking for program /home/ryan/bin/mprun...not found
Checking for program /usr/local/bin/mpiexec...not found
Checking for program /usr/local/bin/mpirun...not found
Checking for program /usr/local/bin/mprun...not found
Checking for program /usr/local/bin/mpiexec...not found
Checking for program /usr/local/bin/mpirun...not found
Checking for program /usr/local/bin/mprun...not found
Checking for program /usr/local/sbin/mpiexec...not found
Checking for program /usr/local/sbin/mpirun...not found
Checking for program /usr/local/sbin/mprun...not found
Checking for program /usr/local/sbin/mpiexec...not found
Checking for program /usr/local/sbin/mpirun...not found
Checking for program /usr/local/sbin/mprun...not found
Checking for program /usr/bin/mpiexec...found
                  Defined make macro "MPIEXEC" to "mpiexec --oversubscribe"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

#ifdef __cplusplus
extern "C"
#endif
int init(int argc,  char *argv[]) {

  int isInitialized;

  MPI_Init(&argc, &argv);
  MPI_Initialized(&isInitialized);
  return (int) isInitialized;

}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/libconftest.so  -shared -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o    -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.MPI/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.MPI/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <mpi.h>

#ifdef __cplusplus
extern "C"
#endif
int checkInit(void) {

  int isInitialized;

  MPI_Initialized(&isInitialized);
  if (isInitialized) MPI_Finalize();
  return (int) isInitialized;

}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.packages.MPI/libconftest.so  -shared -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.packages.MPI/conftest.o    -lstdc++ -ldl 
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"

#include <stdio.h>
#include <stdlib.h>
#ifdef PETSC_HAVE_DLFCN_H
#include <dlfcn.h>
#endif
    

int main() {

  int   argc    = 1;
  char *argv[2] = {(char *) "conftest", NULL};
  void *lib;
  int (*init)(int, char **);
  int (*checkInit)(void);

  lib = dlopen("/tmp/petsc-VuReVk/config.libraries/lib1.so", RTLD_LAZY);
  if (!lib) {
    fprintf(stderr, "Could not open lib1.so: %s\n", dlerror());
    exit(1);
  }
  init = (int (*)(int, char **)) dlsym(lib, "init");
  if (!init) {
    fprintf(stderr, "Could not find initialization function\n");
    exit(1);
  }
  if (!(*init)(argc, argv)) {
    fprintf(stderr, "Could not initialize library\n");
    exit(1);
  }
  lib = dlopen("/tmp/petsc-VuReVk/config.libraries/lib2.so", RTLD_LAZY);
  if (!lib) {
    fprintf(stderr, "Could not open lib2.so: %s\n", dlerror());
    exit(1);
  }
  checkInit = (int (*)(void)) dlsym(lib, "checkInit");
  if (!checkInit) {
    fprintf(stderr, "Could not find initialization check function\n");
    exit(1);
  }
  if (!(*checkInit)()) {
    fprintf(stderr, "Did not link with shared library\n");
    exit(2);
  }
  ;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl  -ldl
Testing executable /tmp/petsc-VuReVk/config.libraries/conftest to see if it can be run
Executing: mpiexec --oversubscribe /tmp/petsc-VuReVk/config.libraries/conftest
Executing: mpiexec --oversubscribe /tmp/petsc-VuReVk/config.libraries/conftest
              Library was shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.yaml(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.yaml(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST configureLibrary from config.packages.valgrind(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:747)
TESTING: configureLibrary from config.packages.valgrind(config/BuildSystem/config/package.py:747)
  Find an installation and check if it can work with PETSc
==================================================================================
            Checking for a functional valgrind
            Not checking for library in Compiler specific search VALGRIND: [] because no functions given to check for
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                No functions to check for in library [] []
                No functions to check for in library [] []
            Checking for headers Compiler specific search VALGRIND: ['/usr/include', '/usr/lib/openmpi']
                Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['valgrind/valgrind.h'] in ['/usr/include', '/usr/lib/openmpi']
                  Checking include with compiler flags var CPPFLAGS ['/usr/include', '/usr/lib/openmpi']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.headers  -I/usr/include -I/usr/lib/openmpi /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/valgrind/valgrind.h" 1 3 4
# 95 "/usr/include/valgrind/valgrind.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 1 3 4
# 40 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 3 4
# 40 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 3 4
typedef __builtin_va_list __gnuc_va_list;
# 99 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdarg.h" 3 4
typedef __gnuc_va_list va_list;
# 96 "/usr/include/valgrind/valgrind.h" 2 3 4
# 399 "/usr/include/valgrind/valgrind.h" 3 4
typedef
   struct {
      unsigned long int nraddr;
   }
   OrigFn;
# 6126 "/usr/include/valgrind/valgrind.h" 3 4
typedef
   enum { VG_USERREQ__RUNNING_ON_VALGRIND = 0x1001,
          VG_USERREQ__DISCARD_TRANSLATIONS = 0x1002,
          VG_USERREQ__CLIENT_CALL0 = 0x1101,
          VG_USERREQ__CLIENT_CALL1 = 0x1102,
          VG_USERREQ__CLIENT_CALL2 = 0x1103,
          VG_USERREQ__CLIENT_CALL3 = 0x1104,
          VG_USERREQ__COUNT_ERRORS = 0x1201,
          VG_USERREQ__GDB_MONITOR_COMMAND = 0x1202,
          VG_USERREQ__MALLOCLIKE_BLOCK = 0x1301,
          VG_USERREQ__RESIZEINPLACE_BLOCK = 0x130b,
          VG_USERREQ__FREELIKE_BLOCK = 0x1302,
          VG_USERREQ__CREATE_MEMPOOL = 0x1303,
          VG_USERREQ__DESTROY_MEMPOOL = 0x1304,
          VG_USERREQ__MEMPOOL_ALLOC = 0x1305,
          VG_USERREQ__MEMPOOL_FREE = 0x1306,
          VG_USERREQ__MEMPOOL_TRIM = 0x1307,
          VG_USERREQ__MOVE_MEMPOOL = 0x1308,
          VG_USERREQ__MEMPOOL_CHANGE = 0x1309,
          VG_USERREQ__MEMPOOL_EXISTS = 0x130a,
# 6171 "/usr/include/valgrind/valgrind.h" 3 4
          VG_USERREQ__PRINTF = 0x1401,
          VG_USERREQ__PRINTF_BACKTRACE = 0x1402,
          VG_USERREQ__PRINTF_VALIST_BY_REF = 0x1403,
          VG_USERREQ__PRINTF_BACKTRACE_VALIST_BY_REF = 0x1404,
          VG_USERREQ__STACK_REGISTER = 0x1501,
          VG_USERREQ__STACK_DEREGISTER = 0x1502,
          VG_USERREQ__STACK_CHANGE = 0x1503,
          VG_USERREQ__LOAD_PDB_DEBUGINFO = 0x1601,
          VG_USERREQ__MAP_IP_TO_SRCLOC = 0x1701,
          VG_USERREQ__CHANGE_ERR_DISABLEMENT = 0x1801,
          VG_USERREQ__VEX_INIT_FOR_IRI = 0x1901,
          VG_USERREQ__INNER_THREADS = 0x1902
   } Vg_ClientRequest;
# 6240 "/usr/include/valgrind/valgrind.h" 3 4
static int VALGRIND_PRINTF(const char *format, ...)
   __attribute__((format(__printf__, 1, 2), __unused__));
static int
VALGRIND_PRINTF(const char *format, ...)
{
   unsigned long _qzz_res;
   va_list vargs;
   __builtin_va_start(vargs,format);
   _qzz_res = __extension__ ({ volatile unsigned long int _zzq_args[6]; volatile unsigned long int _zzq_result; _zzq_args[0] = (unsigned long int)(VG_USERREQ__PRINTF_VALIST_BY_REF); _zzq_args[1] = (unsigned long int)((unsigned long)format); _zzq_args[2] = (unsigned long int)((unsigned long)&vargs); _zzq_args[3] = (unsigned long int)(0); _zzq_args[4] = (unsigned long int)(0); _zzq_args[5] = (unsigned long int)(0); __asm__ volatile("rolq $3,  %%rdi ; rolq $13, %%rdi\n\t" "rolq $61, %%rdi ; rolq $51, %%rdi\n\t" "xchgq %%rbx,%%rbx" : "=d" (_zzq_result) : "a" (&_zzq_args[0]), "0" (0) : "cc", "memory" ); _zzq_result; })
                                      ;
   __builtin_va_end(vargs);
   return (int)_qzz_res;
}
static int VALGRIND_PRINTF_BACKTRACE(const char *format, ...)
   __attribute__((format(__printf__, 1, 2), __unused__));
static int
VALGRIND_PRINTF_BACKTRACE(const char *format, ...)
{
   unsigned long _qzz_res;
   va_list vargs;
   __builtin_va_start(vargs,format);
   _qzz_res = __extension__ ({ volatile unsigned long int _zzq_args[6]; volatile unsigned long int _zzq_result; _zzq_args[0] = (unsigned long int)(VG_USERREQ__PRINTF_BACKTRACE_VALIST_BY_REF); _zzq_args[1] = (unsigned long int)((unsigned long)format); _zzq_args[2] = (unsigned long int)((unsigned long)&vargs); _zzq_args[3] = (unsigned long int)(0); _zzq_args[4] = (unsigned long int)(0); _zzq_args[5] = (unsigned long int)(0); __asm__ volatile("rolq $3,  %%rdi ; rolq $13, %%rdi\n\t" "rolq $61, %%rdi ; rolq $51, %%rdi\n\t" "xchgq %%rbx,%%rbx" : "=d" (_zzq_result) : "a" (&_zzq_args[0]), "0" (0) : "cc", "memory" ); _zzq_result; })
                                      ;
   __builtin_va_end(vargs);
   return (int)_qzz_res;
}
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['valgrind/valgrind.h'] in ['/usr/include', '/usr/lib/openmpi']
                Popping language C
                    All intermediate test results are stored in /tmp/petsc-VuReVk/config.packages.valgrind
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.valgrind/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.valgrind -fPIC -fopenmp   -g -O   /tmp/petsc-VuReVk/config.packages.valgrind/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <valgrind/valgrind.h>

int main() {
RUNNING_ON_VALGRIND;
;
  return 0;
}
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.ssl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.ssl(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.sprng(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.sprng(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
        PETSc clone, checking for Sowing 

Checking for program /home/ryan/bin/pdflatex...not found
Checking for program /usr/local/bin/pdflatex...not found
Checking for program /home/ryan/bin/pdflatex...not found
Checking for program /usr/local/bin/pdflatex...not found
Checking for program /usr/local/sbin/pdflatex...not found
Checking for program /usr/bin/pdflatex...found
            Defined make macro "PDFLATEX" to "/usr/bin/pdflatex"
Checking for program /home/ryan/bin/bfort...not found
Checking for program /usr/local/bin/bfort...not found
Checking for program /home/ryan/bin/bfort...not found
Checking for program /usr/local/bin/bfort...not found
Checking for program /usr/local/sbin/bfort...not found
Checking for program /usr/bin/bfort...not found
Checking for program /usr/bin/site_perl/bfort...not found
Checking for program /usr/bin/vendor_perl/bfort...not found
Checking for program /usr/bin/core_perl/bfort...not found
Checking for program /home/ryan/.local/bin/bfort...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/bfort...not found
Checking for program /home/ryan/bin/doctext...not found
Checking for program /usr/local/bin/doctext...not found
Checking for program /home/ryan/bin/doctext...not found
Checking for program /usr/local/bin/doctext...not found
Checking for program /usr/local/sbin/doctext...not found
Checking for program /usr/bin/doctext...not found
Checking for program /usr/bin/site_perl/doctext...not found
Checking for program /usr/bin/vendor_perl/doctext...not found
Checking for program /usr/bin/core_perl/doctext...not found
Checking for program /home/ryan/.local/bin/doctext...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/doctext...not found
Checking for program /home/ryan/bin/mapnames...not found
Checking for program /usr/local/bin/mapnames...not found
Checking for program /home/ryan/bin/mapnames...not found
Checking for program /usr/local/bin/mapnames...not found
Checking for program /usr/local/sbin/mapnames...not found
Checking for program /usr/bin/mapnames...not found
Checking for program /usr/bin/site_perl/mapnames...not found
Checking for program /usr/bin/vendor_perl/mapnames...not found
Checking for program /usr/bin/core_perl/mapnames...not found
Checking for program /home/ryan/.local/bin/mapnames...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/mapnames...not found
Checking for program /home/ryan/bin/bib2html...not found
Checking for program /usr/local/bin/bib2html...not found
Checking for program /home/ryan/bin/bib2html...not found
Checking for program /usr/local/bin/bib2html...not found
Checking for program /usr/local/sbin/bib2html...not found
Checking for program /usr/bin/bib2html...not found
Checking for program /usr/bin/site_perl/bib2html...not found
Checking for program /usr/bin/vendor_perl/bib2html...not found
Checking for program /usr/bin/core_perl/bib2html...not found
Checking for program /home/ryan/.local/bin/bib2html...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/bib2html...not found
        Bfort not found. Installing sowing for FortranStubs
            Pushing language C
================================================================================
TEST checkDependencies from config.packages.sowing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.sowing(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.sowing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:747)
TESTING: configureLibrary from config.packages.sowing(config/BuildSystem/config/package.py:747)
  Find an installation and check if it can work with PETSc
==================================================================================
              Checking for a functional sowing
                      Looking for SOWING at git.sowing, hg.sowing or a directory starting with ['petsc-pkg-sowing']
                      Found a copy of SOWING in git.sowing
Executing: ['git', 'rev-parse', '--git-dir']
stdout: .git
Executing: ['git', 'cat-file', '-e', 'v1.1.25-p1^{commit}']
Executing: ['git', 'rev-parse', 'v1.1.25-p1']
stdout: bc33abe1e9818d95968d6897bfbfc35ed0c18b51
Executing: ['git', 'stash']
stdout: No local changes to save
Executing: ['git', 'clean', '-f', '-d', '-x']
stdout:
Removing pkg.gitcommit
Removing sowing.petscconf
Executing: ['git', 'checkout', '-f', 'bc33abe1e9818d95968d6897bfbfc35ed0c18b51']
Do not need to rebuild SOWING
              Not checking for library in Download SOWING: [] because no functions given to check for
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                  No functions to check for in library [] []
                  No functions to check for in library [] []
              Checking for headers Download SOWING: ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
================================================================================
TEST checkSharedLibrary from config.packages.sowing(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.sowing(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
            Popping language C
Checking for program /home/ryan/projects/moose/petsc/linux-c-opt/bin/bfort...found
            Defined make macro "BFORT" to "/home/ryan/projects/moose/petsc/linux-c-opt/bin/bfort"
Checking for program /home/ryan/projects/moose/petsc/linux-c-opt/bin/doctext...found
            Defined make macro "DOCTEXT" to "/home/ryan/projects/moose/petsc/linux-c-opt/bin/doctext"
Checking for program /home/ryan/projects/moose/petsc/linux-c-opt/bin/mapnames...found
            Defined make macro "MAPNAMES" to "/home/ryan/projects/moose/petsc/linux-c-opt/bin/mapnames"
Checking for program /home/ryan/projects/moose/petsc/linux-c-opt/bin/bib2html...found
            Defined make macro "BIB2HTML" to "/home/ryan/projects/moose/petsc/linux-c-opt/bin/bib2html"
Executing: /home/ryan/projects/moose/petsc/linux-c-opt/bin/bfort -version
stdout: bfort (sowing) release 1.1.25 of $Date: 2006-02-20 14:58:34 $
           Running /home/ryan/projects/moose/petsc/linux-c-opt/bin/bfort to generate fortran stubs
================================================================================
TEST alternateConfigureLibrary from config.packages.slepc(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/slepc.py:83)
TESTING: alternateConfigureLibrary from config.packages.slepc(config/BuildSystem/config/packages/slepc.py:83)
              Defined make rule "slepc-build" with dependencies "" and code []
              Defined make rule "slepc-install" with dependencies "" and code []
================================================================================
TEST alternateConfigureLibrary from config.packages.revolve(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.revolve(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.radau5(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.radau5(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.pami(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.pami(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.opengles(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.opengles(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.opencl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.opencl(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.muparser(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.muparser(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.mpe(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.mpe(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.memkind(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.memkind(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.libmesh(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/libmesh.py:76)
TESTING: alternateConfigureLibrary from config.packages.libmesh(config/BuildSystem/config/packages/libmesh.py:76)
              Defined make rule "libmesh-build" with dependencies "" and code []
              Defined make rule "libmesh-install" with dependencies "" and code []
================================================================================
TEST alternateConfigureLibrary from config.packages.moose(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.moose(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.libjpeg(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.libjpeg(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.libceed(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.libceed(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
Checking for program /home/ryan/bin/lgrind...not found
Checking for program /usr/local/bin/lgrind...not found
Checking for program /home/ryan/bin/lgrind...not found
Checking for program /usr/local/bin/lgrind...not found
Checking for program /usr/local/sbin/lgrind...not found
Checking for program /usr/bin/lgrind...not found
Checking for program /usr/bin/site_perl/lgrind...not found
Checking for program /usr/bin/vendor_perl/lgrind...not found
Checking for program /usr/bin/core_perl/lgrind...not found
Checking for program /home/ryan/.local/bin/lgrind...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/lgrind...not found
================================================================================
TEST alternateConfigureLibrary from config.packages.gmp(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.gmp(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.mpfr(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.mpfr(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.opengl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.opengl(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.glut(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.glut(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.giflib(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.giflib(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.cuda(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.cuda(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.cusp(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.cusp(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.ctetgen(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.ctetgen(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.concurrencykit(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.concurrencykit(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST locateC2html from config.packages.c2html(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/c2html.py:32)
TESTING: locateC2html from config.packages.c2html(config/BuildSystem/config/packages/c2html.py:32)
Looking for default C2html executable
Checking for program /home/ryan/bin/c2html...not found
Checking for program /usr/local/bin/c2html...not found
Checking for program /home/ryan/bin/c2html...not found
Checking for program /usr/local/bin/c2html...not found
Checking for program /usr/local/sbin/c2html...not found
Checking for program /usr/bin/c2html...not found
Checking for program /usr/bin/site_perl/c2html...not found
Checking for program /usr/bin/vendor_perl/c2html...not found
Checking for program /usr/bin/core_perl/c2html...not found
Checking for program /home/ryan/.local/bin/c2html...not found
Checking for program /home/ryan/projects/moose/petsc/lib/petsc/bin/win32fe/c2html...not found
================================================================================
TEST alternateConfigureLibrary from config.packages.boost(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.boost(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.pthread(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.pthread(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.pthread(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/pthread.py:19)
TESTING: configureLibrary from config.packages.pthread(config/BuildSystem/config/packages/pthread.py:19)
   Checks for pthread_barrier_t, cpu_set_t, and sys/sysctl.h 
==================================================================================
              Checking for a functional pthread
              Checking for library in Compiler specific search PTHREAD: []
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                  Checking for functions [pthread_create] in library [] []
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char pthread_create();
static void _check_pthread_create() { pthread_create(); }

int main() {
_check_pthread_create();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
                    Popping language C
                  No functions to check for in library [] []
              Checking for headers Compiler specific search PTHREAD: ['/usr/include', '/usr/lib/openmpi']
                  Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['pthread.h'] in ['/usr/include', '/usr/lib/openmpi']
                    Checking include with compiler flags var CPPFLAGS ['/usr/include', '/usr/lib/openmpi']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.headers  -I/usr/include -I/usr/lib/openmpi /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/pthread.h" 1 3 4
# 21 "/usr/include/pthread.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 27 "/usr/include/bits/byteswap.h" 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/bits/byteswap.h" 2 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 23 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/sched.h" 1 3 4
# 29 "/usr/include/sched.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 30 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 32 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 33 "/usr/include/sched.h" 2 3 4
typedef __pid_t pid_t;
# 1 "/usr/include/bits/sched.h" 1 3 4
# 74 "/usr/include/bits/sched.h" 3 4
# 1 "/usr/include/bits/types/struct_sched_param.h" 1 3 4
# 23 "/usr/include/bits/types/struct_sched_param.h" 3 4
struct sched_param
{
  int sched_priority;
};
# 75 "/usr/include/bits/sched.h" 2 3 4
# 93 "/usr/include/bits/sched.h" 3 4
# 44 "/usr/include/sched.h" 2 3 4
# 1 "/usr/include/bits/cpu-set.h" 1 3 4
# 32 "/usr/include/bits/cpu-set.h" 3 4
typedef unsigned long int __cpu_mask;
typedef struct
{
  __cpu_mask __bits[1024 / (8 * sizeof (__cpu_mask))];
} cpu_set_t;
# 115 "/usr/include/bits/cpu-set.h" 3 4
extern int __sched_cpucount (size_t __setsize, const cpu_set_t *__setp)
     __attribute__ ((__nothrow__ , __leaf__));
extern cpu_set_t *__sched_cpualloc (size_t __count) __attribute__ ((__nothrow__ , __leaf__)) ;
extern void __sched_cpufree (cpu_set_t *__set) __attribute__ ((__nothrow__ , __leaf__));
# 45 "/usr/include/sched.h" 2 3 4
extern int sched_setparam (__pid_t __pid, const struct sched_param *__param)
     __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getparam (__pid_t __pid, struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_setscheduler (__pid_t __pid, int __policy,
          const struct sched_param *__param) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_getscheduler (__pid_t __pid) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_yield (void) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_max (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_get_priority_min (int __algorithm) __attribute__ ((__nothrow__ , __leaf__));
extern int sched_rr_get_interval (__pid_t __pid, struct timespec *__t) __attribute__ ((__nothrow__ , __leaf__));
# 129 "/usr/include/sched.h" 3 4
# 24 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/time.h" 1 3 4
# 29 "/usr/include/time.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 30 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/time.h" 1 3 4
# 34 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 38 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_tm.h" 1 3 4
struct tm
{
  int tm_sec;
  int tm_min;
  int tm_hour;
  int tm_mday;
  int tm_mon;
  int tm_year;
  int tm_wday;
  int tm_yday;
  int tm_isdst;
  long int tm_gmtoff;
  const char *tm_zone;
};
# 40 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 47 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 48 "/usr/include/time.h" 2 3 4
# 1 "/usr/include/bits/types/struct_itimerspec.h" 1 3 4
struct itimerspec
  {
    struct timespec it_interval;
    struct timespec it_value;
  };
# 49 "/usr/include/time.h" 2 3 4
struct sigevent;
# 60 "/usr/include/time.h" 3 4
# 1 "/usr/include/bits/types/locale_t.h" 1 3 4
# 22 "/usr/include/bits/types/locale_t.h" 3 4
# 1 "/usr/include/bits/types/__locale_t.h" 1 3 4
# 28 "/usr/include/bits/types/__locale_t.h" 3 4
struct __locale_struct
{
  struct __locale_data *__locales[13];
  const unsigned short int *__ctype_b;
  const int *__ctype_tolower;
  const int *__ctype_toupper;
  const char *__names[13];
};
typedef struct __locale_struct *__locale_t;
# 23 "/usr/include/bits/types/locale_t.h" 2 3 4
typedef __locale_t locale_t;
# 61 "/usr/include/time.h" 2 3 4
extern clock_t clock (void) __attribute__ ((__nothrow__ , __leaf__));
extern time_t time (time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern double difftime (time_t __time1, time_t __time0)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern time_t mktime (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern size_t strftime (char *__restrict __s, size_t __maxsize,
   const char *__restrict __format,
   const struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
# 104 "/usr/include/time.h" 3 4
extern size_t strftime_l (char *__restrict __s, size_t __maxsize,
     const char *__restrict __format,
     const struct tm *__restrict __tp,
     locale_t __loc) __attribute__ ((__nothrow__ , __leaf__));
# 119 "/usr/include/time.h" 3 4
extern struct tm *gmtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *gmtime_r (const time_t *__restrict __timer,
       struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern struct tm *localtime_r (const time_t *__restrict __timer,
          struct tm *__restrict __tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime (const struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime (const time_t *__timer) __attribute__ ((__nothrow__ , __leaf__));
extern char *asctime_r (const struct tm *__restrict __tp,
   char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *ctime_r (const time_t *__restrict __timer,
        char *__restrict __buf) __attribute__ ((__nothrow__ , __leaf__));
extern char *__tzname[2];
extern int __daylight;
extern long int __timezone;
extern char *tzname[2];
extern void tzset (void) __attribute__ ((__nothrow__ , __leaf__));
extern int daylight;
extern long int timezone;
extern int stime (const time_t *__when) __attribute__ ((__nothrow__ , __leaf__));
# 196 "/usr/include/time.h" 3 4
extern time_t timegm (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern time_t timelocal (struct tm *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int dysize (int __year) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
# 211 "/usr/include/time.h" 3 4
extern int nanosleep (const struct timespec *__requested_time,
        struct timespec *__remaining);
extern int clock_getres (clockid_t __clock_id, struct timespec *__res) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_gettime (clockid_t __clock_id, struct timespec *__tp) __attribute__ ((__nothrow__ , __leaf__));
extern int clock_settime (clockid_t __clock_id, const struct timespec *__tp)
     __attribute__ ((__nothrow__ , __leaf__));
extern int clock_nanosleep (clockid_t __clock_id, int __flags,
       const struct timespec *__req,
       struct timespec *__rem);
extern int clock_getcpuclockid (pid_t __pid, clockid_t *__clock_id) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_create (clockid_t __clock_id,
    struct sigevent *__restrict __evp,
    timer_t *__restrict __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_delete (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_settime (timer_t __timerid, int __flags,
     const struct itimerspec *__restrict __value,
     struct itimerspec *__restrict __ovalue) __attribute__ ((__nothrow__ , __leaf__));
extern int timer_gettime (timer_t __timerid, struct itimerspec *__value)
     __attribute__ ((__nothrow__ , __leaf__));
extern int timer_getoverrun (timer_t __timerid) __attribute__ ((__nothrow__ , __leaf__));
extern int timespec_get (struct timespec *__ts, int __base)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 307 "/usr/include/time.h" 3 4
# 25 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 27 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/setjmp.h" 1 3 4
# 26 "/usr/include/bits/setjmp.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 27 "/usr/include/bits/setjmp.h" 2 3 4
typedef long int __jmp_buf[8];
# 28 "/usr/include/pthread.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 29 "/usr/include/pthread.h" 2 3 4
enum
{
  PTHREAD_CREATE_JOINABLE,
  PTHREAD_CREATE_DETACHED
};
enum
{
  PTHREAD_MUTEX_TIMED_NP,
  PTHREAD_MUTEX_RECURSIVE_NP,
  PTHREAD_MUTEX_ERRORCHECK_NP,
  PTHREAD_MUTEX_ADAPTIVE_NP
  ,
  PTHREAD_MUTEX_NORMAL = PTHREAD_MUTEX_TIMED_NP,
  PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP,
  PTHREAD_MUTEX_ERRORCHECK = PTHREAD_MUTEX_ERRORCHECK_NP,
  PTHREAD_MUTEX_DEFAULT = PTHREAD_MUTEX_NORMAL
};
enum
{
  PTHREAD_MUTEX_STALLED,
  PTHREAD_MUTEX_STALLED_NP = PTHREAD_MUTEX_STALLED,
  PTHREAD_MUTEX_ROBUST,
  PTHREAD_MUTEX_ROBUST_NP = PTHREAD_MUTEX_ROBUST
};
enum
{
  PTHREAD_PRIO_NONE,
  PTHREAD_PRIO_INHERIT,
  PTHREAD_PRIO_PROTECT
};
# 115 "/usr/include/pthread.h" 3 4
enum
{
  PTHREAD_RWLOCK_PREFER_READER_NP,
  PTHREAD_RWLOCK_PREFER_WRITER_NP,
  PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
  PTHREAD_RWLOCK_DEFAULT_NP = PTHREAD_RWLOCK_PREFER_READER_NP
};
# 156 "/usr/include/pthread.h" 3 4
enum
{
  PTHREAD_INHERIT_SCHED,
  PTHREAD_EXPLICIT_SCHED
};
enum
{
  PTHREAD_SCOPE_SYSTEM,
  PTHREAD_SCOPE_PROCESS
};
enum
{
  PTHREAD_PROCESS_PRIVATE,
  PTHREAD_PROCESS_SHARED
};
# 191 "/usr/include/pthread.h" 3 4
struct _pthread_cleanup_buffer
{
  void (*__routine) (void *);
  void *__arg;
  int __canceltype;
  struct _pthread_cleanup_buffer *__prev;
};
enum
{
  PTHREAD_CANCEL_ENABLE,
  PTHREAD_CANCEL_DISABLE
};
enum
{
  PTHREAD_CANCEL_DEFERRED,
  PTHREAD_CANCEL_ASYNCHRONOUS
};
# 229 "/usr/include/pthread.h" 3 4
extern int pthread_create (pthread_t *__restrict __newthread,
      const pthread_attr_t *__restrict __attr,
      void *(*__start_routine) (void *),
      void *__restrict __arg) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 3)));
extern void pthread_exit (void *__retval) __attribute__ ((__noreturn__));
extern int pthread_join (pthread_t __th, void **__thread_return);
# 272 "/usr/include/pthread.h" 3 4
extern int pthread_detach (pthread_t __th) __attribute__ ((__nothrow__ , __leaf__));
extern pthread_t pthread_self (void) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int pthread_equal (pthread_t __thread1, pthread_t __thread2)
  __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern int pthread_attr_init (pthread_attr_t *__attr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_destroy (pthread_attr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getdetachstate (const pthread_attr_t *__attr,
     int *__detachstate)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setdetachstate (pthread_attr_t *__attr,
     int __detachstate)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getguardsize (const pthread_attr_t *__attr,
          size_t *__guardsize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setguardsize (pthread_attr_t *__attr,
          size_t __guardsize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getschedparam (const pthread_attr_t *__restrict __attr,
           struct sched_param *__restrict __param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setschedparam (pthread_attr_t *__restrict __attr,
           const struct sched_param *__restrict
           __param) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_getschedpolicy (const pthread_attr_t *__restrict
     __attr, int *__restrict __policy)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setschedpolicy (pthread_attr_t *__attr, int __policy)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getinheritsched (const pthread_attr_t *__restrict
      __attr, int *__restrict __inherit)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setinheritsched (pthread_attr_t *__attr,
      int __inherit)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getscope (const pthread_attr_t *__restrict __attr,
      int *__restrict __scope)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setscope (pthread_attr_t *__attr, int __scope)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getstackaddr (const pthread_attr_t *__restrict
          __attr, void **__restrict __stackaddr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2))) __attribute__ ((__deprecated__));
extern int pthread_attr_setstackaddr (pthread_attr_t *__attr,
          void *__stackaddr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1))) __attribute__ ((__deprecated__));
extern int pthread_attr_getstacksize (const pthread_attr_t *__restrict
          __attr, size_t *__restrict __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_attr_setstacksize (pthread_attr_t *__attr,
          size_t __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_attr_getstack (const pthread_attr_t *__restrict __attr,
      void **__restrict __stackaddr,
      size_t *__restrict __stacksize)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2, 3)));
extern int pthread_attr_setstack (pthread_attr_t *__attr, void *__stackaddr,
      size_t __stacksize) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 430 "/usr/include/pthread.h" 3 4
extern int pthread_setschedparam (pthread_t __target_thread, int __policy,
      const struct sched_param *__param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (3)));
extern int pthread_getschedparam (pthread_t __target_thread,
      int *__restrict __policy,
      struct sched_param *__restrict __param)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2, 3)));
extern int pthread_setschedprio (pthread_t __target_thread, int __prio)
     __attribute__ ((__nothrow__ , __leaf__));
# 495 "/usr/include/pthread.h" 3 4
extern int pthread_once (pthread_once_t *__once_control,
    void (*__init_routine) (void)) __attribute__ ((__nonnull__ (1, 2)));
# 507 "/usr/include/pthread.h" 3 4
extern int pthread_setcancelstate (int __state, int *__oldstate);
extern int pthread_setcanceltype (int __type, int *__oldtype);
extern int pthread_cancel (pthread_t __th);
extern void pthread_testcancel (void);
typedef struct
{
  struct
  {
    __jmp_buf __cancel_jmp_buf;
    int __mask_was_saved;
  } __cancel_jmp_buf[1];
  void *__pad[4];
} __pthread_unwind_buf_t __attribute__ ((__aligned__));
# 541 "/usr/include/pthread.h" 3 4
struct __pthread_cleanup_frame
{
  void (*__cancel_routine) (void *);
  void *__cancel_arg;
  int __do_it;
  int __cancel_type;
};
# 681 "/usr/include/pthread.h" 3 4
extern void __pthread_register_cancel (__pthread_unwind_buf_t *__buf)
     ;
# 693 "/usr/include/pthread.h" 3 4
extern void __pthread_unregister_cancel (__pthread_unwind_buf_t *__buf)
  ;
# 734 "/usr/include/pthread.h" 3 4
extern void __pthread_unwind_next (__pthread_unwind_buf_t *__buf)
     __attribute__ ((__noreturn__))
     __attribute__ ((__weak__))
     ;
struct __jmp_buf_tag;
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask) __attribute__ ((__nothrow__));
extern int pthread_mutex_init (pthread_mutex_t *__mutex,
          const pthread_mutexattr_t *__mutexattr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_destroy (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_trylock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_lock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_timedlock (pthread_mutex_t *__restrict __mutex,
        const struct timespec *__restrict
        __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutex_unlock (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutex_getprioceiling (const pthread_mutex_t *
      __restrict __mutex,
      int *__restrict __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutex_setprioceiling (pthread_mutex_t *__restrict __mutex,
      int __prioceiling,
      int *__restrict __old_ceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 3)));
extern int pthread_mutex_consistent (pthread_mutex_t *__mutex)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 807 "/usr/include/pthread.h" 3 4
extern int pthread_mutexattr_init (pthread_mutexattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_destroy (pthread_mutexattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getpshared (const pthread_mutexattr_t *
      __restrict __attr,
      int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setpshared (pthread_mutexattr_t *__attr,
      int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_gettype (const pthread_mutexattr_t *__restrict
          __attr, int *__restrict __kind)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_settype (pthread_mutexattr_t *__attr, int __kind)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getprotocol (const pthread_mutexattr_t *
       __restrict __attr,
       int *__restrict __protocol)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setprotocol (pthread_mutexattr_t *__attr,
       int __protocol)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getprioceiling (const pthread_mutexattr_t *
          __restrict __attr,
          int *__restrict __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setprioceiling (pthread_mutexattr_t *__attr,
          int __prioceiling)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_mutexattr_getrobust (const pthread_mutexattr_t *__attr,
     int *__robustness)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_mutexattr_setrobust (pthread_mutexattr_t *__attr,
     int __robustness)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 889 "/usr/include/pthread.h" 3 4
extern int pthread_rwlock_init (pthread_rwlock_t *__restrict __rwlock,
    const pthread_rwlockattr_t *__restrict
    __attr) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_destroy (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_rdlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_tryrdlock (pthread_rwlock_t *__rwlock)
  __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_timedrdlock (pthread_rwlock_t *__restrict __rwlock,
           const struct timespec *__restrict
           __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlock_wrlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_trywrlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlock_timedwrlock (pthread_rwlock_t *__restrict __rwlock,
           const struct timespec *__restrict
           __abstime) __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlock_unlock (pthread_rwlock_t *__rwlock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_init (pthread_rwlockattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_destroy (pthread_rwlockattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_getpshared (const pthread_rwlockattr_t *
       __restrict __attr,
       int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlockattr_setpshared (pthread_rwlockattr_t *__attr,
       int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_rwlockattr_getkind_np (const pthread_rwlockattr_t *
       __restrict __attr,
       int *__restrict __pref)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_rwlockattr_setkind_np (pthread_rwlockattr_t *__attr,
       int __pref) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_init (pthread_cond_t *__restrict __cond,
         const pthread_condattr_t *__restrict __cond_attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_destroy (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_signal (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_broadcast (pthread_cond_t *__cond)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_cond_wait (pthread_cond_t *__restrict __cond,
         pthread_mutex_t *__restrict __mutex)
     __attribute__ ((__nonnull__ (1, 2)));
# 1001 "/usr/include/pthread.h" 3 4
extern int pthread_cond_timedwait (pthread_cond_t *__restrict __cond,
       pthread_mutex_t *__restrict __mutex,
       const struct timespec *__restrict __abstime)
     __attribute__ ((__nonnull__ (1, 2, 3)));
extern int pthread_condattr_init (pthread_condattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_destroy (pthread_condattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_getpshared (const pthread_condattr_t *
     __restrict __attr,
     int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_condattr_setpshared (pthread_condattr_t *__attr,
     int __pshared) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_condattr_getclock (const pthread_condattr_t *
          __restrict __attr,
          __clockid_t *__restrict __clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_condattr_setclock (pthread_condattr_t *__attr,
          __clockid_t __clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1045 "/usr/include/pthread.h" 3 4
extern int pthread_spin_init (pthread_spinlock_t *__lock, int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_destroy (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_lock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_trylock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_spin_unlock (pthread_spinlock_t *__lock)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_init (pthread_barrier_t *__restrict __barrier,
     const pthread_barrierattr_t *__restrict
     __attr, unsigned int __count)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_destroy (pthread_barrier_t *__barrier)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrier_wait (pthread_barrier_t *__barrier)
     __attribute__ ((__nothrow__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_init (pthread_barrierattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_destroy (pthread_barrierattr_t *__attr)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_barrierattr_getpshared (const pthread_barrierattr_t *
        __restrict __attr,
        int *__restrict __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
extern int pthread_barrierattr_setpshared (pthread_barrierattr_t *__attr,
        int __pshared)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
# 1112 "/usr/include/pthread.h" 3 4
extern int pthread_key_create (pthread_key_t *__key,
          void (*__destr_function) (void *))
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));
extern int pthread_key_delete (pthread_key_t __key) __attribute__ ((__nothrow__ , __leaf__));
extern void *pthread_getspecific (pthread_key_t __key) __attribute__ ((__nothrow__ , __leaf__));
extern int pthread_setspecific (pthread_key_t __key,
    const void *__pointer) __attribute__ ((__nothrow__ , __leaf__)) ;
extern int pthread_getcpuclockid (pthread_t __thread_id,
      __clockid_t *__clock_id)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (2)));
# 1146 "/usr/include/pthread.h" 3 4
extern int pthread_atfork (void (*__prepare) (void),
      void (*__parent) (void),
      void (*__child) (void)) __attribute__ ((__nothrow__ , __leaf__));
# 1160 "/usr/include/pthread.h" 3 4
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['pthread.h'] in ['/usr/include', '/usr/lib/openmpi']
                  Popping language C
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.packages.pthread
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.pthread/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.packages.pthread/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <pthread.h>

int main() {
pthread_barrier_t *a;
;
  return 0;
}
              Defined "HAVE_PTHREAD_BARRIER_T" to "1"
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.pthread/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.pthread -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.packages.pthread/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
#include <sched.h>

int main() {
cpu_set_t *a;
;
  return 0;
}
              Defined "HAVE_SCHED_CPU_SET_T" to "1"
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.headers  /tmp/petsc-VuReVk/config.packages.pthread/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.packages.pthread/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.packages.pthread/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.packages.pthread/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.packages.pthread/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.packages.pthread/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.packages.pthread/conftest.c" 2
# 1 "/usr/include/sys/sysctl.h" 1 3 4
# 21 "/usr/include/sys/sysctl.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 22 "/usr/include/sys/sysctl.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 24 "/usr/include/sys/sysctl.h" 2 3 4
# 43 "/usr/include/sys/sysctl.h" 3 4
# 1 "/usr/include/linux/sysctl.h" 1 3 4
# 26 "/usr/include/linux/sysctl.h" 3 4
# 1 "/usr/include/linux/kernel.h" 1 3 4
# 27 "/usr/include/linux/sysctl.h" 2 3 4
# 1 "/usr/include/linux/types.h" 1 3 4
# 28 "/usr/include/linux/sysctl.h" 2 3 4
struct __sysctl_args {
 int *name;
 int nlen;
 void *oldval;
 size_t *oldlenp;
 void *newval;
 size_t newlen;
 unsigned long __unused[4];
};
enum
{
 CTL_KERN=1,
 CTL_VM=2,
 CTL_NET=3,
 CTL_PROC=4,
 CTL_FS=5,
 CTL_DEBUG=6,
 CTL_DEV=7,
 CTL_BUS=8,
 CTL_ABI=9,
 CTL_CPU=10,
 CTL_ARLAN=254,
 CTL_S390DBF=5677,
 CTL_SUNRPC=7249,
 CTL_PM=9899,
 CTL_FRV=9898,
};
enum
{
 CTL_BUS_ISA=1
};
enum
{
 INOTIFY_MAX_USER_INSTANCES=1,
 INOTIFY_MAX_USER_WATCHES=2,
 INOTIFY_MAX_QUEUED_EVENTS=3
};
enum
{
 KERN_OSTYPE=1,
 KERN_OSRELEASE=2,
 KERN_OSREV=3,
 KERN_VERSION=4,
 KERN_SECUREMASK=5,
 KERN_PROF=6,
 KERN_NODENAME=7,
 KERN_DOMAINNAME=8,
 KERN_PANIC=15,
 KERN_REALROOTDEV=16,
 KERN_SPARC_REBOOT=21,
 KERN_CTLALTDEL=22,
 KERN_PRINTK=23,
 KERN_NAMETRANS=24,
 KERN_PPC_HTABRECLAIM=25,
 KERN_PPC_ZEROPAGED=26,
 KERN_PPC_POWERSAVE_NAP=27,
 KERN_MODPROBE=28,
 KERN_SG_BIG_BUFF=29,
 KERN_ACCT=30,
 KERN_PPC_L2CR=31,
 KERN_RTSIGNR=32,
 KERN_RTSIGMAX=33,
 KERN_SHMMAX=34,
 KERN_MSGMAX=35,
 KERN_MSGMNB=36,
 KERN_MSGPOOL=37,
 KERN_SYSRQ=38,
 KERN_MAX_THREADS=39,
  KERN_RANDOM=40,
  KERN_SHMALL=41,
  KERN_MSGMNI=42,
  KERN_SEM=43,
  KERN_SPARC_STOP_A=44,
  KERN_SHMMNI=45,
 KERN_OVERFLOWUID=46,
 KERN_OVERFLOWGID=47,
 KERN_SHMPATH=48,
 KERN_HOTPLUG=49,
 KERN_IEEE_EMULATION_WARNINGS=50,
 KERN_S390_USER_DEBUG_LOGGING=51,
 KERN_CORE_USES_PID=52,
 KERN_TAINTED=53,
 KERN_CADPID=54,
 KERN_PIDMAX=55,
   KERN_CORE_PATTERN=56,
 KERN_PANIC_ON_OOPS=57,
 KERN_HPPA_PWRSW=58,
 KERN_HPPA_UNALIGNED=59,
 KERN_PRINTK_RATELIMIT=60,
 KERN_PRINTK_RATELIMIT_BURST=61,
 KERN_PTY=62,
 KERN_NGROUPS_MAX=63,
 KERN_SPARC_SCONS_PWROFF=64,
 KERN_HZ_TIMER=65,
 KERN_UNKNOWN_NMI_PANIC=66,
 KERN_BOOTLOADER_TYPE=67,
 KERN_RANDOMIZE=68,
 KERN_SETUID_DUMPABLE=69,
 KERN_SPIN_RETRY=70,
 KERN_ACPI_VIDEO_FLAGS=71,
 KERN_IA64_UNALIGNED=72,
 KERN_COMPAT_LOG=73,
 KERN_MAX_LOCK_DEPTH=74,
 KERN_NMI_WATCHDOG=75,
 KERN_PANIC_ON_NMI=76,
 KERN_PANIC_ON_WARN=77,
};
enum
{
 VM_UNUSED1=1,
 VM_UNUSED2=2,
 VM_UNUSED3=3,
 VM_UNUSED4=4,
 VM_OVERCOMMIT_MEMORY=5,
 VM_UNUSED5=6,
 VM_UNUSED7=7,
 VM_UNUSED8=8,
 VM_UNUSED9=9,
 VM_PAGE_CLUSTER=10,
 VM_DIRTY_BACKGROUND=11,
 VM_DIRTY_RATIO=12,
 VM_DIRTY_WB_CS=13,
 VM_DIRTY_EXPIRE_CS=14,
 VM_NR_PDFLUSH_THREADS=15,
 VM_OVERCOMMIT_RATIO=16,
 VM_PAGEBUF=17,
 VM_HUGETLB_PAGES=18,
 VM_SWAPPINESS=19,
 VM_LOWMEM_RESERVE_RATIO=20,
 VM_MIN_FREE_KBYTES=21,
 VM_MAX_MAP_COUNT=22,
 VM_LAPTOP_MODE=23,
 VM_BLOCK_DUMP=24,
 VM_HUGETLB_GROUP=25,
 VM_VFS_CACHE_PRESSURE=26,
 VM_LEGACY_VA_LAYOUT=27,
 VM_SWAP_TOKEN_TIMEOUT=28,
 VM_DROP_PAGECACHE=29,
 VM_PERCPU_PAGELIST_FRACTION=30,
 VM_ZONE_RECLAIM_MODE=31,
 VM_MIN_UNMAPPED=32,
 VM_PANIC_ON_OOM=33,
 VM_VDSO_ENABLED=34,
 VM_MIN_SLAB=35,
};
enum
{
 NET_CORE=1,
 NET_ETHER=2,
 NET_802=3,
 NET_UNIX=4,
 NET_IPV4=5,
 NET_IPX=6,
 NET_ATALK=7,
 NET_NETROM=8,
 NET_AX25=9,
 NET_BRIDGE=10,
 NET_ROSE=11,
 NET_IPV6=12,
 NET_X25=13,
 NET_TR=14,
 NET_DECNET=15,
 NET_ECONET=16,
 NET_SCTP=17,
 NET_LLC=18,
 NET_NETFILTER=19,
 NET_DCCP=20,
 NET_IRDA=412,
};
enum
{
 RANDOM_POOLSIZE=1,
 RANDOM_ENTROPY_COUNT=2,
 RANDOM_READ_THRESH=3,
 RANDOM_WRITE_THRESH=4,
 RANDOM_BOOT_ID=5,
 RANDOM_UUID=6
};
enum
{
 PTY_MAX=1,
 PTY_NR=2
};
enum
{
 BUS_ISA_MEM_BASE=1,
 BUS_ISA_PORT_BASE=2,
 BUS_ISA_PORT_SHIFT=3
};
enum
{
 NET_CORE_WMEM_MAX=1,
 NET_CORE_RMEM_MAX=2,
 NET_CORE_WMEM_DEFAULT=3,
 NET_CORE_RMEM_DEFAULT=4,
 NET_CORE_MAX_BACKLOG=6,
 NET_CORE_FASTROUTE=7,
 NET_CORE_MSG_COST=8,
 NET_CORE_MSG_BURST=9,
 NET_CORE_OPTMEM_MAX=10,
 NET_CORE_HOT_LIST_LENGTH=11,
 NET_CORE_DIVERT_VERSION=12,
 NET_CORE_NO_CONG_THRESH=13,
 NET_CORE_NO_CONG=14,
 NET_CORE_LO_CONG=15,
 NET_CORE_MOD_CONG=16,
 NET_CORE_DEV_WEIGHT=17,
 NET_CORE_SOMAXCONN=18,
 NET_CORE_BUDGET=19,
 NET_CORE_AEVENT_ETIME=20,
 NET_CORE_AEVENT_RSEQTH=21,
 NET_CORE_WARNINGS=22,
};
enum
{
 NET_UNIX_DESTROY_DELAY=1,
 NET_UNIX_DELETE_DELAY=2,
 NET_UNIX_MAX_DGRAM_QLEN=3,
};
enum
{
 NET_NF_CONNTRACK_MAX=1,
 NET_NF_CONNTRACK_TCP_TIMEOUT_SYN_SENT=2,
 NET_NF_CONNTRACK_TCP_TIMEOUT_SYN_RECV=3,
 NET_NF_CONNTRACK_TCP_TIMEOUT_ESTABLISHED=4,
 NET_NF_CONNTRACK_TCP_TIMEOUT_FIN_WAIT=5,
 NET_NF_CONNTRACK_TCP_TIMEOUT_CLOSE_WAIT=6,
 NET_NF_CONNTRACK_TCP_TIMEOUT_LAST_ACK=7,
 NET_NF_CONNTRACK_TCP_TIMEOUT_TIME_WAIT=8,
 NET_NF_CONNTRACK_TCP_TIMEOUT_CLOSE=9,
 NET_NF_CONNTRACK_UDP_TIMEOUT=10,
 NET_NF_CONNTRACK_UDP_TIMEOUT_STREAM=11,
 NET_NF_CONNTRACK_ICMP_TIMEOUT=12,
 NET_NF_CONNTRACK_GENERIC_TIMEOUT=13,
 NET_NF_CONNTRACK_BUCKETS=14,
 NET_NF_CONNTRACK_LOG_INVALID=15,
 NET_NF_CONNTRACK_TCP_TIMEOUT_MAX_RETRANS=16,
 NET_NF_CONNTRACK_TCP_LOOSE=17,
 NET_NF_CONNTRACK_TCP_BE_LIBERAL=18,
 NET_NF_CONNTRACK_TCP_MAX_RETRANS=19,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_CLOSED=20,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_COOKIE_WAIT=21,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_COOKIE_ECHOED=22,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_ESTABLISHED=23,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_SENT=24,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_RECD=25,
 NET_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_ACK_SENT=26,
 NET_NF_CONNTRACK_COUNT=27,
 NET_NF_CONNTRACK_ICMPV6_TIMEOUT=28,
 NET_NF_CONNTRACK_FRAG6_TIMEOUT=29,
 NET_NF_CONNTRACK_FRAG6_LOW_THRESH=30,
 NET_NF_CONNTRACK_FRAG6_HIGH_THRESH=31,
 NET_NF_CONNTRACK_CHECKSUM=32,
};
enum
{
 NET_IPV4_FORWARD=8,
 NET_IPV4_DYNADDR=9,
 NET_IPV4_CONF=16,
 NET_IPV4_NEIGH=17,
 NET_IPV4_ROUTE=18,
 NET_IPV4_FIB_HASH=19,
 NET_IPV4_NETFILTER=20,
 NET_IPV4_TCP_TIMESTAMPS=33,
 NET_IPV4_TCP_WINDOW_SCALING=34,
 NET_IPV4_TCP_SACK=35,
 NET_IPV4_TCP_RETRANS_COLLAPSE=36,
 NET_IPV4_DEFAULT_TTL=37,
 NET_IPV4_AUTOCONFIG=38,
 NET_IPV4_NO_PMTU_DISC=39,
 NET_IPV4_TCP_SYN_RETRIES=40,
 NET_IPV4_IPFRAG_HIGH_THRESH=41,
 NET_IPV4_IPFRAG_LOW_THRESH=42,
 NET_IPV4_IPFRAG_TIME=43,
 NET_IPV4_TCP_MAX_KA_PROBES=44,
 NET_IPV4_TCP_KEEPALIVE_TIME=45,
 NET_IPV4_TCP_KEEPALIVE_PROBES=46,
 NET_IPV4_TCP_RETRIES1=47,
 NET_IPV4_TCP_RETRIES2=48,
 NET_IPV4_TCP_FIN_TIMEOUT=49,
 NET_IPV4_IP_MASQ_DEBUG=50,
 NET_TCP_SYNCOOKIES=51,
 NET_TCP_STDURG=52,
 NET_TCP_RFC1337=53,
 NET_TCP_SYN_TAILDROP=54,
 NET_TCP_MAX_SYN_BACKLOG=55,
 NET_IPV4_LOCAL_PORT_RANGE=56,
 NET_IPV4_ICMP_ECHO_IGNORE_ALL=57,
 NET_IPV4_ICMP_ECHO_IGNORE_BROADCASTS=58,
 NET_IPV4_ICMP_SOURCEQUENCH_RATE=59,
 NET_IPV4_ICMP_DESTUNREACH_RATE=60,
 NET_IPV4_ICMP_TIMEEXCEED_RATE=61,
 NET_IPV4_ICMP_PARAMPROB_RATE=62,
 NET_IPV4_ICMP_ECHOREPLY_RATE=63,
 NET_IPV4_ICMP_IGNORE_BOGUS_ERROR_RESPONSES=64,
 NET_IPV4_IGMP_MAX_MEMBERSHIPS=65,
 NET_TCP_TW_RECYCLE=66,
 NET_IPV4_ALWAYS_DEFRAG=67,
 NET_IPV4_TCP_KEEPALIVE_INTVL=68,
 NET_IPV4_INET_PEER_THRESHOLD=69,
 NET_IPV4_INET_PEER_MINTTL=70,
 NET_IPV4_INET_PEER_MAXTTL=71,
 NET_IPV4_INET_PEER_GC_MINTIME=72,
 NET_IPV4_INET_PEER_GC_MAXTIME=73,
 NET_TCP_ORPHAN_RETRIES=74,
 NET_TCP_ABORT_ON_OVERFLOW=75,
 NET_TCP_SYNACK_RETRIES=76,
 NET_TCP_MAX_ORPHANS=77,
 NET_TCP_MAX_TW_BUCKETS=78,
 NET_TCP_FACK=79,
 NET_TCP_REORDERING=80,
 NET_TCP_ECN=81,
 NET_TCP_DSACK=82,
 NET_TCP_MEM=83,
 NET_TCP_WMEM=84,
 NET_TCP_RMEM=85,
 NET_TCP_APP_WIN=86,
 NET_TCP_ADV_WIN_SCALE=87,
 NET_IPV4_NONLOCAL_BIND=88,
 NET_IPV4_ICMP_RATELIMIT=89,
 NET_IPV4_ICMP_RATEMASK=90,
 NET_TCP_TW_REUSE=91,
 NET_TCP_FRTO=92,
 NET_TCP_LOW_LATENCY=93,
 NET_IPV4_IPFRAG_SECRET_INTERVAL=94,
 NET_IPV4_IGMP_MAX_MSF=96,
 NET_TCP_NO_METRICS_SAVE=97,
 NET_TCP_DEFAULT_WIN_SCALE=105,
 NET_TCP_MODERATE_RCVBUF=106,
 NET_TCP_TSO_WIN_DIVISOR=107,
 NET_TCP_BIC_BETA=108,
 NET_IPV4_ICMP_ERRORS_USE_INBOUND_IFADDR=109,
 NET_TCP_CONG_CONTROL=110,
 NET_TCP_ABC=111,
 NET_IPV4_IPFRAG_MAX_DIST=112,
  NET_TCP_MTU_PROBING=113,
 NET_TCP_BASE_MSS=114,
 NET_IPV4_TCP_WORKAROUND_SIGNED_WINDOWS=115,
 NET_TCP_DMA_COPYBREAK=116,
 NET_TCP_SLOW_START_AFTER_IDLE=117,
 NET_CIPSOV4_CACHE_ENABLE=118,
 NET_CIPSOV4_CACHE_BUCKET_SIZE=119,
 NET_CIPSOV4_RBM_OPTFMT=120,
 NET_CIPSOV4_RBM_STRICTVALID=121,
 NET_TCP_AVAIL_CONG_CONTROL=122,
 NET_TCP_ALLOWED_CONG_CONTROL=123,
 NET_TCP_MAX_SSTHRESH=124,
 NET_TCP_FRTO_RESPONSE=125,
};
enum {
 NET_IPV4_ROUTE_FLUSH=1,
 NET_IPV4_ROUTE_MIN_DELAY=2,
 NET_IPV4_ROUTE_MAX_DELAY=3,
 NET_IPV4_ROUTE_GC_THRESH=4,
 NET_IPV4_ROUTE_MAX_SIZE=5,
 NET_IPV4_ROUTE_GC_MIN_INTERVAL=6,
 NET_IPV4_ROUTE_GC_TIMEOUT=7,
 NET_IPV4_ROUTE_GC_INTERVAL=8,
 NET_IPV4_ROUTE_REDIRECT_LOAD=9,
 NET_IPV4_ROUTE_REDIRECT_NUMBER=10,
 NET_IPV4_ROUTE_REDIRECT_SILENCE=11,
 NET_IPV4_ROUTE_ERROR_COST=12,
 NET_IPV4_ROUTE_ERROR_BURST=13,
 NET_IPV4_ROUTE_GC_ELASTICITY=14,
 NET_IPV4_ROUTE_MTU_EXPIRES=15,
 NET_IPV4_ROUTE_MIN_PMTU=16,
 NET_IPV4_ROUTE_MIN_ADVMSS=17,
 NET_IPV4_ROUTE_SECRET_INTERVAL=18,
 NET_IPV4_ROUTE_GC_MIN_INTERVAL_MS=19,
};
enum
{
 NET_PROTO_CONF_ALL=-2,
 NET_PROTO_CONF_DEFAULT=-3
};
enum
{
 NET_IPV4_CONF_FORWARDING=1,
 NET_IPV4_CONF_MC_FORWARDING=2,
 NET_IPV4_CONF_PROXY_ARP=3,
 NET_IPV4_CONF_ACCEPT_REDIRECTS=4,
 NET_IPV4_CONF_SECURE_REDIRECTS=5,
 NET_IPV4_CONF_SEND_REDIRECTS=6,
 NET_IPV4_CONF_SHARED_MEDIA=7,
 NET_IPV4_CONF_RP_FILTER=8,
 NET_IPV4_CONF_ACCEPT_SOURCE_ROUTE=9,
 NET_IPV4_CONF_BOOTP_RELAY=10,
 NET_IPV4_CONF_LOG_MARTIANS=11,
 NET_IPV4_CONF_TAG=12,
 NET_IPV4_CONF_ARPFILTER=13,
 NET_IPV4_CONF_MEDIUM_ID=14,
 NET_IPV4_CONF_NOXFRM=15,
 NET_IPV4_CONF_NOPOLICY=16,
 NET_IPV4_CONF_FORCE_IGMP_VERSION=17,
 NET_IPV4_CONF_ARP_ANNOUNCE=18,
 NET_IPV4_CONF_ARP_IGNORE=19,
 NET_IPV4_CONF_PROMOTE_SECONDARIES=20,
 NET_IPV4_CONF_ARP_ACCEPT=21,
 NET_IPV4_CONF_ARP_NOTIFY=22,
};
enum
{
 NET_IPV4_NF_CONNTRACK_MAX=1,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_SYN_SENT=2,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_SYN_RECV=3,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_ESTABLISHED=4,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_FIN_WAIT=5,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_CLOSE_WAIT=6,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_LAST_ACK=7,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_TIME_WAIT=8,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_CLOSE=9,
 NET_IPV4_NF_CONNTRACK_UDP_TIMEOUT=10,
 NET_IPV4_NF_CONNTRACK_UDP_TIMEOUT_STREAM=11,
 NET_IPV4_NF_CONNTRACK_ICMP_TIMEOUT=12,
 NET_IPV4_NF_CONNTRACK_GENERIC_TIMEOUT=13,
 NET_IPV4_NF_CONNTRACK_BUCKETS=14,
 NET_IPV4_NF_CONNTRACK_LOG_INVALID=15,
 NET_IPV4_NF_CONNTRACK_TCP_TIMEOUT_MAX_RETRANS=16,
 NET_IPV4_NF_CONNTRACK_TCP_LOOSE=17,
 NET_IPV4_NF_CONNTRACK_TCP_BE_LIBERAL=18,
 NET_IPV4_NF_CONNTRACK_TCP_MAX_RETRANS=19,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_CLOSED=20,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_COOKIE_WAIT=21,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_COOKIE_ECHOED=22,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_ESTABLISHED=23,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_SENT=24,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_RECD=25,
  NET_IPV4_NF_CONNTRACK_SCTP_TIMEOUT_SHUTDOWN_ACK_SENT=26,
 NET_IPV4_NF_CONNTRACK_COUNT=27,
 NET_IPV4_NF_CONNTRACK_CHECKSUM=28,
};
enum {
 NET_IPV6_CONF=16,
 NET_IPV6_NEIGH=17,
 NET_IPV6_ROUTE=18,
 NET_IPV6_ICMP=19,
 NET_IPV6_BINDV6ONLY=20,
 NET_IPV6_IP6FRAG_HIGH_THRESH=21,
 NET_IPV6_IP6FRAG_LOW_THRESH=22,
 NET_IPV6_IP6FRAG_TIME=23,
 NET_IPV6_IP6FRAG_SECRET_INTERVAL=24,
 NET_IPV6_MLD_MAX_MSF=25,
};
enum {
 NET_IPV6_ROUTE_FLUSH=1,
 NET_IPV6_ROUTE_GC_THRESH=2,
 NET_IPV6_ROUTE_MAX_SIZE=3,
 NET_IPV6_ROUTE_GC_MIN_INTERVAL=4,
 NET_IPV6_ROUTE_GC_TIMEOUT=5,
 NET_IPV6_ROUTE_GC_INTERVAL=6,
 NET_IPV6_ROUTE_GC_ELASTICITY=7,
 NET_IPV6_ROUTE_MTU_EXPIRES=8,
 NET_IPV6_ROUTE_MIN_ADVMSS=9,
 NET_IPV6_ROUTE_GC_MIN_INTERVAL_MS=10
};
enum {
 NET_IPV6_FORWARDING=1,
 NET_IPV6_HOP_LIMIT=2,
 NET_IPV6_MTU=3,
 NET_IPV6_ACCEPT_RA=4,
 NET_IPV6_ACCEPT_REDIRECTS=5,
 NET_IPV6_AUTOCONF=6,
 NET_IPV6_DAD_TRANSMITS=7,
 NET_IPV6_RTR_SOLICITS=8,
 NET_IPV6_RTR_SOLICIT_INTERVAL=9,
 NET_IPV6_RTR_SOLICIT_DELAY=10,
 NET_IPV6_USE_TEMPADDR=11,
 NET_IPV6_TEMP_VALID_LFT=12,
 NET_IPV6_TEMP_PREFERED_LFT=13,
 NET_IPV6_REGEN_MAX_RETRY=14,
 NET_IPV6_MAX_DESYNC_FACTOR=15,
 NET_IPV6_MAX_ADDRESSES=16,
 NET_IPV6_FORCE_MLD_VERSION=17,
 NET_IPV6_ACCEPT_RA_DEFRTR=18,
 NET_IPV6_ACCEPT_RA_PINFO=19,
 NET_IPV6_ACCEPT_RA_RTR_PREF=20,
 NET_IPV6_RTR_PROBE_INTERVAL=21,
 NET_IPV6_ACCEPT_RA_RT_INFO_MAX_PLEN=22,
 NET_IPV6_PROXY_NDP=23,
 NET_IPV6_ACCEPT_SOURCE_ROUTE=25,
 NET_IPV6_ACCEPT_RA_FROM_LOCAL=26,
 NET_IPV6_ACCEPT_RA_RT_INFO_MIN_PLEN=27,
 __NET_IPV6_MAX
};
enum {
 NET_IPV6_ICMP_RATELIMIT=1
};
enum {
 NET_NEIGH_MCAST_SOLICIT=1,
 NET_NEIGH_UCAST_SOLICIT=2,
 NET_NEIGH_APP_SOLICIT=3,
 NET_NEIGH_RETRANS_TIME=4,
 NET_NEIGH_REACHABLE_TIME=5,
 NET_NEIGH_DELAY_PROBE_TIME=6,
 NET_NEIGH_GC_STALE_TIME=7,
 NET_NEIGH_UNRES_QLEN=8,
 NET_NEIGH_PROXY_QLEN=9,
 NET_NEIGH_ANYCAST_DELAY=10,
 NET_NEIGH_PROXY_DELAY=11,
 NET_NEIGH_LOCKTIME=12,
 NET_NEIGH_GC_INTERVAL=13,
 NET_NEIGH_GC_THRESH1=14,
 NET_NEIGH_GC_THRESH2=15,
 NET_NEIGH_GC_THRESH3=16,
 NET_NEIGH_RETRANS_TIME_MS=17,
 NET_NEIGH_REACHABLE_TIME_MS=18,
};
enum {
 NET_DCCP_DEFAULT=1,
};
enum {
 NET_IPX_PPROP_BROADCASTING=1,
 NET_IPX_FORWARDING=2
};
enum {
 NET_LLC2=1,
 NET_LLC_STATION=2,
};
enum {
 NET_LLC2_TIMEOUT=1,
};
enum {
 NET_LLC_STATION_ACK_TIMEOUT=1,
};
enum {
 NET_LLC2_ACK_TIMEOUT=1,
 NET_LLC2_P_TIMEOUT=2,
 NET_LLC2_REJ_TIMEOUT=3,
 NET_LLC2_BUSY_TIMEOUT=4,
};
enum {
 NET_ATALK_AARP_EXPIRY_TIME=1,
 NET_ATALK_AARP_TICK_TIME=2,
 NET_ATALK_AARP_RETRANSMIT_LIMIT=3,
 NET_ATALK_AARP_RESOLVE_TIME=4
};
enum {
 NET_NETROM_DEFAULT_PATH_QUALITY=1,
 NET_NETROM_OBSOLESCENCE_COUNT_INITIALISER=2,
 NET_NETROM_NETWORK_TTL_INITIALISER=3,
 NET_NETROM_TRANSPORT_TIMEOUT=4,
 NET_NETROM_TRANSPORT_MAXIMUM_TRIES=5,
 NET_NETROM_TRANSPORT_ACKNOWLEDGE_DELAY=6,
 NET_NETROM_TRANSPORT_BUSY_DELAY=7,
 NET_NETROM_TRANSPORT_REQUESTED_WINDOW_SIZE=8,
 NET_NETROM_TRANSPORT_NO_ACTIVITY_TIMEOUT=9,
 NET_NETROM_ROUTING_CONTROL=10,
 NET_NETROM_LINK_FAILS_COUNT=11,
 NET_NETROM_RESET=12
};
enum {
 NET_AX25_IP_DEFAULT_MODE=1,
 NET_AX25_DEFAULT_MODE=2,
 NET_AX25_BACKOFF_TYPE=3,
 NET_AX25_CONNECT_MODE=4,
 NET_AX25_STANDARD_WINDOW=5,
 NET_AX25_EXTENDED_WINDOW=6,
 NET_AX25_T1_TIMEOUT=7,
 NET_AX25_T2_TIMEOUT=8,
 NET_AX25_T3_TIMEOUT=9,
 NET_AX25_IDLE_TIMEOUT=10,
 NET_AX25_N2=11,
 NET_AX25_PACLEN=12,
 NET_AX25_PROTOCOL=13,
 NET_AX25_DAMA_SLAVE_TIMEOUT=14
};
enum {
 NET_ROSE_RESTART_REQUEST_TIMEOUT=1,
 NET_ROSE_CALL_REQUEST_TIMEOUT=2,
 NET_ROSE_RESET_REQUEST_TIMEOUT=3,
 NET_ROSE_CLEAR_REQUEST_TIMEOUT=4,
 NET_ROSE_ACK_HOLD_BACK_TIMEOUT=5,
 NET_ROSE_ROUTING_CONTROL=6,
 NET_ROSE_LINK_FAIL_TIMEOUT=7,
 NET_ROSE_MAX_VCS=8,
 NET_ROSE_WINDOW_SIZE=9,
 NET_ROSE_NO_ACTIVITY_TIMEOUT=10
};
enum {
 NET_X25_RESTART_REQUEST_TIMEOUT=1,
 NET_X25_CALL_REQUEST_TIMEOUT=2,
 NET_X25_RESET_REQUEST_TIMEOUT=3,
 NET_X25_CLEAR_REQUEST_TIMEOUT=4,
 NET_X25_ACK_HOLD_BACK_TIMEOUT=5,
 NET_X25_FORWARD=6
};
enum
{
 NET_TR_RIF_TIMEOUT=1
};
enum {
 NET_DECNET_NODE_TYPE = 1,
 NET_DECNET_NODE_ADDRESS = 2,
 NET_DECNET_NODE_NAME = 3,
 NET_DECNET_DEFAULT_DEVICE = 4,
 NET_DECNET_TIME_WAIT = 5,
 NET_DECNET_DN_COUNT = 6,
 NET_DECNET_DI_COUNT = 7,
 NET_DECNET_DR_COUNT = 8,
 NET_DECNET_DST_GC_INTERVAL = 9,
 NET_DECNET_CONF = 10,
 NET_DECNET_NO_FC_MAX_CWND = 11,
 NET_DECNET_MEM = 12,
 NET_DECNET_RMEM = 13,
 NET_DECNET_WMEM = 14,
 NET_DECNET_DEBUG_LEVEL = 255
};
enum {
 NET_DECNET_CONF_LOOPBACK = -2,
 NET_DECNET_CONF_DDCMP = -3,
 NET_DECNET_CONF_PPP = -4,
 NET_DECNET_CONF_X25 = -5,
 NET_DECNET_CONF_GRE = -6,
 NET_DECNET_CONF_ETHER = -7
};
enum {
 NET_DECNET_CONF_DEV_PRIORITY = 1,
 NET_DECNET_CONF_DEV_T1 = 2,
 NET_DECNET_CONF_DEV_T2 = 3,
 NET_DECNET_CONF_DEV_T3 = 4,
 NET_DECNET_CONF_DEV_FORWARDING = 5,
 NET_DECNET_CONF_DEV_BLKSIZE = 6,
 NET_DECNET_CONF_DEV_STATE = 7
};
enum {
 NET_SCTP_RTO_INITIAL = 1,
 NET_SCTP_RTO_MIN = 2,
 NET_SCTP_RTO_MAX = 3,
 NET_SCTP_RTO_ALPHA = 4,
 NET_SCTP_RTO_BETA = 5,
 NET_SCTP_VALID_COOKIE_LIFE = 6,
 NET_SCTP_ASSOCIATION_MAX_RETRANS = 7,
 NET_SCTP_PATH_MAX_RETRANS = 8,
 NET_SCTP_MAX_INIT_RETRANSMITS = 9,
 NET_SCTP_HB_INTERVAL = 10,
 NET_SCTP_PRESERVE_ENABLE = 11,
 NET_SCTP_MAX_BURST = 12,
 NET_SCTP_ADDIP_ENABLE = 13,
 NET_SCTP_PRSCTP_ENABLE = 14,
 NET_SCTP_SNDBUF_POLICY = 15,
 NET_SCTP_SACK_TIMEOUT = 16,
 NET_SCTP_RCVBUF_POLICY = 17,
};
enum {
 NET_BRIDGE_NF_CALL_ARPTABLES = 1,
 NET_BRIDGE_NF_CALL_IPTABLES = 2,
 NET_BRIDGE_NF_CALL_IP6TABLES = 3,
 NET_BRIDGE_NF_FILTER_VLAN_TAGGED = 4,
 NET_BRIDGE_NF_FILTER_PPPOE_TAGGED = 5,
};
enum
{
 FS_NRINODE=1,
 FS_STATINODE=2,
 FS_MAXINODE=3,
 FS_NRDQUOT=4,
 FS_MAXDQUOT=5,
 FS_NRFILE=6,
 FS_MAXFILE=7,
 FS_DENTRY=8,
 FS_NRSUPER=9,
 FS_MAXSUPER=10,
 FS_OVERFLOWUID=11,
 FS_OVERFLOWGID=12,
 FS_LEASES=13,
 FS_DIR_NOTIFY=14,
 FS_LEASE_TIME=15,
 FS_DQSTATS=16,
 FS_XFS=17,
 FS_AIO_NR=18,
 FS_AIO_MAX_NR=19,
 FS_INOTIFY=20,
 FS_OCFS2=988,
};
enum {
 FS_DQ_LOOKUPS = 1,
 FS_DQ_DROPS = 2,
 FS_DQ_READS = 3,
 FS_DQ_WRITES = 4,
 FS_DQ_CACHE_HITS = 5,
 FS_DQ_ALLOCATED = 6,
 FS_DQ_FREE = 7,
 FS_DQ_SYNCS = 8,
 FS_DQ_WARNINGS = 9,
};
enum {
 DEV_CDROM=1,
 DEV_HWMON=2,
 DEV_PARPORT=3,
 DEV_RAID=4,
 DEV_MAC_HID=5,
 DEV_SCSI=6,
 DEV_IPMI=7,
};
enum {
 DEV_CDROM_INFO=1,
 DEV_CDROM_AUTOCLOSE=2,
 DEV_CDROM_AUTOEJECT=3,
 DEV_CDROM_DEBUG=4,
 DEV_CDROM_LOCK=5,
 DEV_CDROM_CHECK_MEDIA=6
};
enum {
 DEV_PARPORT_DEFAULT=-3
};
enum {
 DEV_RAID_SPEED_LIMIT_MIN=1,
 DEV_RAID_SPEED_LIMIT_MAX=2
};
enum {
 DEV_PARPORT_DEFAULT_TIMESLICE=1,
 DEV_PARPORT_DEFAULT_SPINTIME=2
};
enum {
 DEV_PARPORT_SPINTIME=1,
 DEV_PARPORT_BASE_ADDR=2,
 DEV_PARPORT_IRQ=3,
 DEV_PARPORT_DMA=4,
 DEV_PARPORT_MODES=5,
 DEV_PARPORT_DEVICES=6,
 DEV_PARPORT_AUTOPROBE=16
};
enum {
 DEV_PARPORT_DEVICES_ACTIVE=-3,
};
enum {
 DEV_PARPORT_DEVICE_TIMESLICE=1,
};
enum {
 DEV_MAC_HID_KEYBOARD_SENDS_LINUX_KEYCODES=1,
 DEV_MAC_HID_KEYBOARD_LOCK_KEYCODES=2,
 DEV_MAC_HID_MOUSE_BUTTON_EMULATION=3,
 DEV_MAC_HID_MOUSE_BUTTON2_KEYCODE=4,
 DEV_MAC_HID_MOUSE_BUTTON3_KEYCODE=5,
 DEV_MAC_HID_ADB_MOUSE_SENDS_KEYCODES=6
};
enum {
 DEV_SCSI_LOGGING_LEVEL=1,
};
enum {
 DEV_IPMI_POWEROFF_POWERCYCLE=1,
};
enum
{
 ABI_DEFHANDLER_COFF=1,
 ABI_DEFHANDLER_ELF=2,
 ABI_DEFHANDLER_LCALL7=3,
 ABI_DEFHANDLER_LIBCSO=4,
 ABI_TRACE=5,
 ABI_FAKE_UTSNAME=6,
};
# 44 "/usr/include/sys/sysctl.h" 2 3 4
# 63 "/usr/include/sys/sysctl.h" 3 4
# 1 "/usr/include/bits/sysctl.h" 1 3 4
# 64 "/usr/include/sys/sysctl.h" 2 3 4
extern int sysctl (int *__name, int __nlen, void *__oldval,
     size_t *__oldlenp, void *__newval, size_t __newlen) __attribute__ ((__nothrow__ , __leaf__));
# 3 "/tmp/petsc-VuReVk/config.packages.pthread/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
              Defined "HAVE_SYS_SYSCTL_H" to "1"
================================================================================
TEST checkSharedLibrary from config.packages.pthread(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.pthread(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.hwloc(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.hwloc(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.openmp(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.openmp(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.viennacl(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.viennacl(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.X(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.X(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.X(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:747)
TESTING: configureLibrary from config.packages.X(config/BuildSystem/config/package.py:747)
  Find an installation and check if it can work with PETSc
==================================================================================
            Checking for a functional X
            Checking for library in Compiler specific search X: []
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                Checking for functions [XSetWMName] in library [] []
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char XSetWMName();
static void _check_XSetWMName() { XSetWMName(); }

int main() {
_check_XSetWMName();;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: /tmp/petsc-VuReVk/config.libraries/conftest.o: in function `_check_XSetWMName':
/tmp/petsc-VuReVk/config.libraries/conftest.c:5: undefined reference to `XSetWMName'
collect2: error: ld returned 1 exit status
                  Popping language C
            Checking for library in Compiler specific search X: ['libX11.a']
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                Checking for functions [XSetWMName] in library ['libX11.a'] []
                  Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char XSetWMName();
static void _check_XSetWMName() { XSetWMName(); }

int main() {
_check_XSetWMName();;
  return 0;
}
                          Pushing language C
                          Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -lX11 -lstdc++ -ldl 
                  Defined "HAVE_LIBX11" to "1"
                  Popping language C
                No functions to check for in library ['libX11.a'] []
            Checking for headers Compiler specific search X: ['/usr/include', '/usr/lib/openmpi']
                Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['X11/Xlib.h'] in ['/usr/include', '/usr/lib/openmpi']
                  Checking include with compiler flags var CPPFLAGS ['/usr/include', '/usr/lib/openmpi']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.headers  -I/usr/include -I/usr/lib/openmpi /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/usr/include/X11/Xlib.h" 1 3 4
# 38 "/usr/include/X11/Xlib.h" 3 4
# 1 "/usr/include/sys/types.h" 1 3 4
# 25 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 30 "/usr/include/sys/types.h" 2 3 4
typedef __u_char u_char;
typedef __u_short u_short;
typedef __u_int u_int;
typedef __u_long u_long;
typedef __quad_t quad_t;
typedef __u_quad_t u_quad_t;
typedef __fsid_t fsid_t;
typedef __loff_t loff_t;
typedef __ino_t ino_t;
# 59 "/usr/include/sys/types.h" 3 4
typedef __dev_t dev_t;
typedef __gid_t gid_t;
typedef __mode_t mode_t;
typedef __nlink_t nlink_t;
typedef __uid_t uid_t;
typedef __off_t off_t;
# 97 "/usr/include/sys/types.h" 3 4
typedef __pid_t pid_t;
typedef __id_t id_t;
typedef __ssize_t ssize_t;
typedef __daddr_t daddr_t;
typedef __caddr_t caddr_t;
typedef __key_t key_t;
# 1 "/usr/include/bits/types/clock_t.h" 1 3 4
typedef __clock_t clock_t;
# 127 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/clockid_t.h" 1 3 4
typedef __clockid_t clockid_t;
# 129 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/time_t.h" 1 3 4
typedef __time_t time_t;
# 130 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/bits/types/timer_t.h" 1 3 4
typedef __timer_t timer_t;
# 131 "/usr/include/sys/types.h" 2 3 4
# 144 "/usr/include/sys/types.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 145 "/usr/include/sys/types.h" 2 3 4
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 156 "/usr/include/sys/types.h" 2 3 4
# 177 "/usr/include/sys/types.h" 3 4
typedef unsigned int u_int8_t __attribute__ ((__mode__ (__QI__)));
typedef unsigned int u_int16_t __attribute__ ((__mode__ (__HI__)));
typedef unsigned int u_int32_t __attribute__ ((__mode__ (__SI__)));
typedef unsigned int u_int64_t __attribute__ ((__mode__ (__DI__)));
typedef int register_t __attribute__ ((__mode__ (__word__)));
# 193 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/endian.h" 1 3 4
# 36 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/endian.h" 1 3 4
# 37 "/usr/include/endian.h" 2 3 4
# 60 "/usr/include/endian.h" 3 4
# 1 "/usr/include/bits/byteswap.h" 1 3 4
# 33 "/usr/include/bits/byteswap.h" 3 4
static __inline __uint16_t
__bswap_16 (__uint16_t __bsx)
{
  return __builtin_bswap16 (__bsx);
}
static __inline __uint32_t
__bswap_32 (__uint32_t __bsx)
{
  return __builtin_bswap32 (__bsx);
}
# 69 "/usr/include/bits/byteswap.h" 3 4
__extension__ static __inline __uint64_t
__bswap_64 (__uint64_t __bsx)
{
  return __builtin_bswap64 (__bsx);
}
# 61 "/usr/include/endian.h" 2 3 4
# 1 "/usr/include/bits/uintn-identity.h" 1 3 4
# 32 "/usr/include/bits/uintn-identity.h" 3 4
static __inline __uint16_t
__uint16_identity (__uint16_t __x)
{
  return __x;
}
static __inline __uint32_t
__uint32_identity (__uint32_t __x)
{
  return __x;
}
static __inline __uint64_t
__uint64_identity (__uint64_t __x)
{
  return __x;
}
# 62 "/usr/include/endian.h" 2 3 4
# 194 "/usr/include/sys/types.h" 2 3 4
# 1 "/usr/include/sys/select.h" 1 3 4
# 30 "/usr/include/sys/select.h" 3 4
# 1 "/usr/include/bits/select.h" 1 3 4
# 22 "/usr/include/bits/select.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 23 "/usr/include/bits/select.h" 2 3 4
# 31 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/sigset_t.h" 1 3 4
# 1 "/usr/include/bits/types/__sigset_t.h" 1 3 4
typedef struct
{
  unsigned long int __val[(1024 / (8 * sizeof (unsigned long int)))];
} __sigset_t;
# 5 "/usr/include/bits/types/sigset_t.h" 2 3 4
typedef __sigset_t sigset_t;
# 34 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timeval.h" 1 3 4
struct timeval
{
  __time_t tv_sec;
  __suseconds_t tv_usec;
};
# 38 "/usr/include/sys/select.h" 2 3 4
# 1 "/usr/include/bits/types/struct_timespec.h" 1 3 4
# 9 "/usr/include/bits/types/struct_timespec.h" 3 4
struct timespec
{
  __time_t tv_sec;
  __syscall_slong_t tv_nsec;
};
# 40 "/usr/include/sys/select.h" 2 3 4
typedef __suseconds_t suseconds_t;
typedef long int __fd_mask;
# 59 "/usr/include/sys/select.h" 3 4
typedef struct
  {
    __fd_mask __fds_bits[1024 / (8 * (int) sizeof (__fd_mask))];
  } fd_set;
typedef __fd_mask fd_mask;
# 91 "/usr/include/sys/select.h" 3 4
# 101 "/usr/include/sys/select.h" 3 4
extern int select (int __nfds, fd_set *__restrict __readfds,
     fd_set *__restrict __writefds,
     fd_set *__restrict __exceptfds,
     struct timeval *__restrict __timeout);
# 113 "/usr/include/sys/select.h" 3 4
extern int pselect (int __nfds, fd_set *__restrict __readfds,
      fd_set *__restrict __writefds,
      fd_set *__restrict __exceptfds,
      const struct timespec *__restrict __timeout,
      const __sigset_t *__restrict __sigmask);
# 126 "/usr/include/sys/select.h" 3 4
# 197 "/usr/include/sys/types.h" 2 3 4
typedef __blksize_t blksize_t;
typedef __blkcnt_t blkcnt_t;
typedef __fsblkcnt_t fsblkcnt_t;
typedef __fsfilcnt_t fsfilcnt_t;
# 244 "/usr/include/sys/types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes.h" 1 3 4
# 23 "/usr/include/bits/pthreadtypes.h" 3 4
# 1 "/usr/include/bits/thread-shared-types.h" 1 3 4
# 77 "/usr/include/bits/thread-shared-types.h" 3 4
# 1 "/usr/include/bits/pthreadtypes-arch.h" 1 3 4
# 21 "/usr/include/bits/pthreadtypes-arch.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 22 "/usr/include/bits/pthreadtypes-arch.h" 2 3 4
# 65 "/usr/include/bits/pthreadtypes-arch.h" 3 4
struct __pthread_rwlock_arch_t
{
  unsigned int __readers;
  unsigned int __writers;
  unsigned int __wrphase_futex;
  unsigned int __writers_futex;
  unsigned int __pad3;
  unsigned int __pad4;
  int __cur_writer;
  int __shared;
  signed char __rwelision;
  unsigned char __pad1[7];
  unsigned long int __pad2;
  unsigned int __flags;
# 99 "/usr/include/bits/pthreadtypes-arch.h" 3 4
};
# 78 "/usr/include/bits/thread-shared-types.h" 2 3 4
typedef struct __pthread_internal_list
{
  struct __pthread_internal_list *__prev;
  struct __pthread_internal_list *__next;
} __pthread_list_t;
# 118 "/usr/include/bits/thread-shared-types.h" 3 4
struct __pthread_mutex_s
{
  int __lock ;
  unsigned int __count;
  int __owner;
  unsigned int __nusers;
  int __kind;
 
  short __spins; short __elision;
  __pthread_list_t __list;
# 145 "/usr/include/bits/thread-shared-types.h" 3 4
 
};
struct __pthread_cond_s
{
  __extension__ union
  {
    __extension__ unsigned long long int __wseq;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __wseq32;
  };
  __extension__ union
  {
    __extension__ unsigned long long int __g1_start;
    struct
    {
      unsigned int __low;
      unsigned int __high;
    } __g1_start32;
  };
  unsigned int __g_refs[2] ;
  unsigned int __g_size[2];
  unsigned int __g1_orig_size;
  unsigned int __wrefs;
  unsigned int __g_signals[2];
};
# 24 "/usr/include/bits/pthreadtypes.h" 2 3 4
typedef unsigned long int pthread_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_mutexattr_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_condattr_t;
typedef unsigned int pthread_key_t;
typedef int pthread_once_t;
union pthread_attr_t
{
  char __size[56];
  long int __align;
};
typedef union pthread_attr_t pthread_attr_t;
typedef union
{
  struct __pthread_mutex_s __data;
  char __size[40];
  long int __align;
} pthread_mutex_t;
typedef union
{
  struct __pthread_cond_s __data;
  char __size[48];
  __extension__ long long int __align;
} pthread_cond_t;
typedef union
{
  struct __pthread_rwlock_arch_t __data;
  char __size[56];
  long int __align;
} pthread_rwlock_t;
typedef union
{
  char __size[8];
  long int __align;
} pthread_rwlockattr_t;
typedef volatile int pthread_spinlock_t;
typedef union
{
  char __size[32];
  long int __align;
} pthread_barrier_t;
typedef union
{
  char __size[4];
  int __align;
} pthread_barrierattr_t;
# 245 "/usr/include/sys/types.h" 2 3 4
# 39 "/usr/include/X11/Xlib.h" 2 3 4
# 1 "/usr/include/X11/X.h" 1 3 4
# 66 "/usr/include/X11/X.h" 3 4
typedef unsigned long XID;
typedef unsigned long Mask;
typedef unsigned long Atom;
typedef unsigned long VisualID;
typedef unsigned long Time;
# 96 "/usr/include/X11/X.h" 3 4
typedef XID Window;
typedef XID Drawable;
typedef XID Font;
typedef XID Pixmap;
typedef XID Cursor;
typedef XID Colormap;
typedef XID GContext;
typedef XID KeySym;
typedef unsigned char KeyCode;
# 45 "/usr/include/X11/Xlib.h" 2 3 4
# 1 "/usr/include/X11/Xfuncproto.h" 1 3 4
# 48 "/usr/include/X11/Xlib.h" 2 3 4
# 1 "/usr/include/X11/Xosdefs.h" 1 3 4
# 49 "/usr/include/X11/Xlib.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long int ptrdiff_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef struct {
  long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
  long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
# 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
} max_align_t;
# 52 "/usr/include/X11/Xlib.h" 2 3 4
# 62 "/usr/include/X11/Xlib.h" 3 4
extern int
_Xmblen(
    char *str,
    int len
    );
# 80 "/usr/include/X11/Xlib.h" 3 4
typedef char *XPointer;
# 148 "/usr/include/X11/Xlib.h" 3 4
typedef struct _XExtData {
 int number;
 struct _XExtData *next;
 int (*free_private)(
 struct _XExtData *extension
 );
 XPointer private_data;
} XExtData;
typedef struct {
 int extension;
 int major_opcode;
 int first_event;
 int first_error;
} XExtCodes;
typedef struct {
    int depth;
    int bits_per_pixel;
    int scanline_pad;
} XPixmapFormatValues;
typedef struct {
 int function;
 unsigned long plane_mask;
 unsigned long foreground;
 unsigned long background;
 int line_width;
 int line_style;
 int cap_style;
 int join_style;
 int fill_style;
 int fill_rule;
 int arc_mode;
 Pixmap tile;
 Pixmap stipple;
 int ts_x_origin;
 int ts_y_origin;
        Font font;
 int subwindow_mode;
 int graphics_exposures;
 int clip_x_origin;
 int clip_y_origin;
 Pixmap clip_mask;
 int dash_offset;
 char dashes;
} XGCValues;
typedef struct _XGC
*GC;
typedef struct {
 XExtData *ext_data;
 VisualID visualid;
 int class;
 unsigned long red_mask, green_mask, blue_mask;
 int bits_per_rgb;
 int map_entries;
} Visual;
typedef struct {
 int depth;
 int nvisuals;
 Visual *visuals;
} Depth;
struct _XDisplay;
typedef struct {
 XExtData *ext_data;
 struct _XDisplay *display;
 Window root;
 int width, height;
 int mwidth, mheight;
 int ndepths;
 Depth *depths;
 int root_depth;
 Visual *root_visual;
 GC default_gc;
 Colormap cmap;
 unsigned long white_pixel;
 unsigned long black_pixel;
 int max_maps, min_maps;
 int backing_store;
 int save_unders;
 long root_input_mask;
} Screen;
typedef struct {
 XExtData *ext_data;
 int depth;
 int bits_per_pixel;
 int scanline_pad;
} ScreenFormat;
typedef struct {
    Pixmap background_pixmap;
    unsigned long background_pixel;
    Pixmap border_pixmap;
    unsigned long border_pixel;
    int bit_gravity;
    int win_gravity;
    int backing_store;
    unsigned long backing_planes;
    unsigned long backing_pixel;
    int save_under;
    long event_mask;
    long do_not_propagate_mask;
    int override_redirect;
    Colormap colormap;
    Cursor cursor;
} XSetWindowAttributes;
typedef struct {
    int x, y;
    int width, height;
    int border_width;
    int depth;
    Visual *visual;
    Window root;
    int class;
    int bit_gravity;
    int win_gravity;
    int backing_store;
    unsigned long backing_planes;
    unsigned long backing_pixel;
    int save_under;
    Colormap colormap;
    int map_installed;
    int map_state;
    long all_event_masks;
    long your_event_mask;
    long do_not_propagate_mask;
    int override_redirect;
    Screen *screen;
} XWindowAttributes;
typedef struct {
 int family;
 int length;
 char *address;
} XHostAddress;
typedef struct {
 int typelength;
 int valuelength;
 char *type;
 char *value;
} XServerInterpretedAddress;
typedef struct _XImage {
    int width, height;
    int xoffset;
    int format;
    char *data;
    int byte_order;
    int bitmap_unit;
    int bitmap_bit_order;
    int bitmap_pad;
    int depth;
    int bytes_per_line;
    int bits_per_pixel;
    unsigned long red_mask;
    unsigned long green_mask;
    unsigned long blue_mask;
    XPointer obdata;
    struct funcs {
 struct _XImage *(*create_image)(
  struct _XDisplay* ,
  Visual* ,
  unsigned int ,
  int ,
  int ,
  char* ,
  unsigned int ,
  unsigned int ,
  int ,
  int );
 int (*destroy_image) (struct _XImage *);
 unsigned long (*get_pixel) (struct _XImage *, int, int);
 int (*put_pixel) (struct _XImage *, int, int, unsigned long);
 struct _XImage *(*sub_image)(struct _XImage *, int, int, unsigned int, unsigned int);
 int (*add_pixel) (struct _XImage *, long);
 } f;
} XImage;
typedef struct {
    int x, y;
    int width, height;
    int border_width;
    Window sibling;
    int stack_mode;
} XWindowChanges;
typedef struct {
 unsigned long pixel;
 unsigned short red, green, blue;
 char flags;
 char pad;
} XColor;
typedef struct {
    short x1, y1, x2, y2;
} XSegment;
typedef struct {
    short x, y;
} XPoint;
typedef struct {
    short x, y;
    unsigned short width, height;
} XRectangle;
typedef struct {
    short x, y;
    unsigned short width, height;
    short angle1, angle2;
} XArc;
typedef struct {
        int key_click_percent;
        int bell_percent;
        int bell_pitch;
        int bell_duration;
        int led;
        int led_mode;
        int key;
        int auto_repeat_mode;
} XKeyboardControl;
typedef struct {
        int key_click_percent;
 int bell_percent;
 unsigned int bell_pitch, bell_duration;
 unsigned long led_mask;
 int global_auto_repeat;
 char auto_repeats[32];
} XKeyboardState;
typedef struct {
        Time time;
 short x, y;
} XTimeCoord;
typedef struct {
  int max_keypermod;
  KeyCode *modifiermap;
} XModifierKeymap;
# 487 "/usr/include/X11/Xlib.h" 3 4
typedef struct _XDisplay Display;
struct _XPrivate;
struct _XrmHashBucketRec;
typedef struct
{
 XExtData *ext_data;
 struct _XPrivate *private1;
 int fd;
 int private2;
 int proto_major_version;
 int proto_minor_version;
 char *vendor;
        XID private3;
 XID private4;
 XID private5;
 int private6;
 XID (*resource_alloc)(
  struct _XDisplay*
 );
 int byte_order;
 int bitmap_unit;
 int bitmap_pad;
 int bitmap_bit_order;
 int nformats;
 ScreenFormat *pixmap_format;
 int private8;
 int release;
 struct _XPrivate *private9, *private10;
 int qlen;
 unsigned long last_request_read;
 unsigned long request;
 XPointer private11;
 XPointer private12;
 XPointer private13;
 XPointer private14;
 unsigned max_request_size;
 struct _XrmHashBucketRec *db;
 int (*private15)(
  struct _XDisplay*
  );
 char *display_name;
 int default_screen;
 int nscreens;
 Screen *screens;
 unsigned long motion_buffer;
 unsigned long private16;
 int min_keycode;
 int max_keycode;
 XPointer private17;
 XPointer private18;
 int private19;
 char *xdefaults;
}
*_XPrivDisplay;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Window root;
 Window subwindow;
 Time time;
 int x, y;
 int x_root, y_root;
 unsigned int state;
 unsigned int keycode;
 int same_screen;
} XKeyEvent;
typedef XKeyEvent XKeyPressedEvent;
typedef XKeyEvent XKeyReleasedEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Window root;
 Window subwindow;
 Time time;
 int x, y;
 int x_root, y_root;
 unsigned int state;
 unsigned int button;
 int same_screen;
} XButtonEvent;
typedef XButtonEvent XButtonPressedEvent;
typedef XButtonEvent XButtonReleasedEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Window root;
 Window subwindow;
 Time time;
 int x, y;
 int x_root, y_root;
 unsigned int state;
 char is_hint;
 int same_screen;
} XMotionEvent;
typedef XMotionEvent XPointerMovedEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Window root;
 Window subwindow;
 Time time;
 int x, y;
 int x_root, y_root;
 int mode;
 int detail;
 int same_screen;
 int focus;
 unsigned int state;
} XCrossingEvent;
typedef XCrossingEvent XEnterWindowEvent;
typedef XCrossingEvent XLeaveWindowEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 int mode;
 int detail;
} XFocusChangeEvent;
typedef XFocusChangeEvent XFocusInEvent;
typedef XFocusChangeEvent XFocusOutEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 char key_vector[32];
} XKeymapEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 int x, y;
 int width, height;
 int count;
} XExposeEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Drawable drawable;
 int x, y;
 int width, height;
 int count;
 int major_code;
 int minor_code;
} XGraphicsExposeEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Drawable drawable;
 int major_code;
 int minor_code;
} XNoExposeEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 int state;
} XVisibilityEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window parent;
 Window window;
 int x, y;
 int width, height;
 int border_width;
 int override_redirect;
} XCreateWindowEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
} XDestroyWindowEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 int from_configure;
} XUnmapEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 int override_redirect;
} XMapEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window parent;
 Window window;
} XMapRequestEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 Window parent;
 int x, y;
 int override_redirect;
} XReparentEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 int x, y;
 int width, height;
 int border_width;
 Window above;
 int override_redirect;
} XConfigureEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 int x, y;
} XGravityEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 int width, height;
} XResizeRequestEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window parent;
 Window window;
 int x, y;
 int width, height;
 int border_width;
 Window above;
 int detail;
 unsigned long value_mask;
} XConfigureRequestEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window event;
 Window window;
 int place;
} XCirculateEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window parent;
 Window window;
 int place;
} XCirculateRequestEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Atom atom;
 Time time;
 int state;
} XPropertyEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Atom selection;
 Time time;
} XSelectionClearEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window owner;
 Window requestor;
 Atom selection;
 Atom target;
 Atom property;
 Time time;
} XSelectionRequestEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window requestor;
 Atom selection;
 Atom target;
 Atom property;
 Time time;
} XSelectionEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Colormap colormap;
 int new;
 int state;
} XColormapEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 Atom message_type;
 int format;
 union {
  char b[20];
  short s[10];
  long l[5];
  } data;
} XClientMessageEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
 int request;
 int first_keycode;
 int count;
} XMappingEvent;
typedef struct {
 int type;
 Display *display;
 XID resourceid;
 unsigned long serial;
 unsigned char error_code;
 unsigned char request_code;
 unsigned char minor_code;
} XErrorEvent;
typedef struct {
 int type;
 unsigned long serial;
 int send_event;
 Display *display;
 Window window;
} XAnyEvent;
typedef struct
    {
    int type;
    unsigned long serial;
    int send_event;
    Display *display;
    int extension;
    int evtype;
    } XGenericEvent;
typedef struct {
    int type;
    unsigned long serial;
    int send_event;
    Display *display;
    int extension;
    int evtype;
    unsigned int cookie;
    void *data;
} XGenericEventCookie;
typedef union _XEvent {
        int type;
 XAnyEvent xany;
 XKeyEvent xkey;
 XButtonEvent xbutton;
 XMotionEvent xmotion;
 XCrossingEvent xcrossing;
 XFocusChangeEvent xfocus;
 XExposeEvent xexpose;
 XGraphicsExposeEvent xgraphicsexpose;
 XNoExposeEvent xnoexpose;
 XVisibilityEvent xvisibility;
 XCreateWindowEvent xcreatewindow;
 XDestroyWindowEvent xdestroywindow;
 XUnmapEvent xunmap;
 XMapEvent xmap;
 XMapRequestEvent xmaprequest;
 XReparentEvent xreparent;
 XConfigureEvent xconfigure;
 XGravityEvent xgravity;
 XResizeRequestEvent xresizerequest;
 XConfigureRequestEvent xconfigurerequest;
 XCirculateEvent xcirculate;
 XCirculateRequestEvent xcirculaterequest;
 XPropertyEvent xproperty;
 XSelectionClearEvent xselectionclear;
 XSelectionRequestEvent xselectionrequest;
 XSelectionEvent xselection;
 XColormapEvent xcolormap;
 XClientMessageEvent xclient;
 XMappingEvent xmapping;
 XErrorEvent xerror;
 XKeymapEvent xkeymap;
 XGenericEvent xgeneric;
 XGenericEventCookie xcookie;
 long pad[24];
} XEvent;
typedef struct {
    short lbearing;
    short rbearing;
    short width;
    short ascent;
    short descent;
    unsigned short attributes;
} XCharStruct;
typedef struct {
    Atom name;
    unsigned long card32;
} XFontProp;
typedef struct {
    XExtData *ext_data;
    Font fid;
    unsigned direction;
    unsigned min_char_or_byte2;
    unsigned max_char_or_byte2;
    unsigned min_byte1;
    unsigned max_byte1;
    int all_chars_exist;
    unsigned default_char;
    int n_properties;
    XFontProp *properties;
    XCharStruct min_bounds;
    XCharStruct max_bounds;
    XCharStruct *per_char;
    int ascent;
    int descent;
} XFontStruct;
typedef struct {
    char *chars;
    int nchars;
    int delta;
    Font font;
} XTextItem;
typedef struct {
    unsigned char byte1;
    unsigned char byte2;
} XChar2b;
typedef struct {
    XChar2b *chars;
    int nchars;
    int delta;
    Font font;
} XTextItem16;
typedef union { Display *display;
  GC gc;
  Visual *visual;
  Screen *screen;
  ScreenFormat *pixmap_format;
  XFontStruct *font; } XEDataObject;
typedef struct {
    XRectangle max_ink_extent;
    XRectangle max_logical_extent;
} XFontSetExtents;
typedef struct _XOM *XOM;
typedef struct _XOC *XOC, *XFontSet;
typedef struct {
    char *chars;
    int nchars;
    int delta;
    XFontSet font_set;
} XmbTextItem;
typedef struct {
    wchar_t *chars;
    int nchars;
    int delta;
    XFontSet font_set;
} XwcTextItem;
# 1121 "/usr/include/X11/Xlib.h" 3 4
typedef struct {
    int charset_count;
    char **charset_list;
} XOMCharSetList;
typedef enum {
    XOMOrientation_LTR_TTB,
    XOMOrientation_RTL_TTB,
    XOMOrientation_TTB_LTR,
    XOMOrientation_TTB_RTL,
    XOMOrientation_Context
} XOrientation;
typedef struct {
    int num_orientation;
    XOrientation *orientation;
} XOMOrientation;
typedef struct {
    int num_font;
    XFontStruct **font_struct_list;
    char **font_name_list;
} XOMFontInfo;
typedef struct _XIM *XIM;
typedef struct _XIC *XIC;
typedef void (*XIMProc)(
    XIM,
    XPointer,
    XPointer
);
typedef int (*XICProc)(
    XIC,
    XPointer,
    XPointer
);
typedef void (*XIDProc)(
    Display*,
    XPointer,
    XPointer
);
typedef unsigned long XIMStyle;
typedef struct {
    unsigned short count_styles;
    XIMStyle *supported_styles;
} XIMStyles;
# 1233 "/usr/include/X11/Xlib.h" 3 4
typedef void *XVaNestedList;
typedef struct {
    XPointer client_data;
    XIMProc callback;
} XIMCallback;
typedef struct {
    XPointer client_data;
    XICProc callback;
} XICCallback;
typedef unsigned long XIMFeedback;
# 1257 "/usr/include/X11/Xlib.h" 3 4
typedef struct _XIMText {
    unsigned short length;
    XIMFeedback *feedback;
    int encoding_is_wchar;
    union {
 char *multi_byte;
 wchar_t *wide_char;
    } string;
} XIMText;
typedef unsigned long XIMPreeditState;
typedef struct _XIMPreeditStateNotifyCallbackStruct {
    XIMPreeditState state;
} XIMPreeditStateNotifyCallbackStruct;
typedef unsigned long XIMResetState;
typedef unsigned long XIMStringConversionFeedback;
# 1291 "/usr/include/X11/Xlib.h" 3 4
typedef struct _XIMStringConversionText {
    unsigned short length;
    XIMStringConversionFeedback *feedback;
    int encoding_is_wchar;
    union {
 char *mbs;
 wchar_t *wcs;
    } string;
} XIMStringConversionText;
typedef unsigned short XIMStringConversionPosition;
typedef unsigned short XIMStringConversionType;
typedef unsigned short XIMStringConversionOperation;
typedef enum {
    XIMForwardChar, XIMBackwardChar,
    XIMForwardWord, XIMBackwardWord,
    XIMCaretUp, XIMCaretDown,
    XIMNextLine, XIMPreviousLine,
    XIMLineStart, XIMLineEnd,
    XIMAbsolutePosition,
    XIMDontChange
} XIMCaretDirection;
typedef struct _XIMStringConversionCallbackStruct {
    XIMStringConversionPosition position;
    XIMCaretDirection direction;
    XIMStringConversionOperation operation;
    unsigned short factor;
    XIMStringConversionText *text;
} XIMStringConversionCallbackStruct;
typedef struct _XIMPreeditDrawCallbackStruct {
    int caret;
    int chg_first;
    int chg_length;
    XIMText *text;
} XIMPreeditDrawCallbackStruct;
typedef enum {
    XIMIsInvisible,
    XIMIsPrimary,
    XIMIsSecondary
} XIMCaretStyle;
typedef struct _XIMPreeditCaretCallbackStruct {
    int position;
    XIMCaretDirection direction;
    XIMCaretStyle style;
} XIMPreeditCaretCallbackStruct;
typedef enum {
    XIMTextType,
    XIMBitmapType
} XIMStatusDataType;
typedef struct _XIMStatusDrawCallbackStruct {
    XIMStatusDataType type;
    union {
 XIMText *text;
 Pixmap bitmap;
    } data;
} XIMStatusDrawCallbackStruct;
typedef struct _XIMHotKeyTrigger {
    KeySym keysym;
    int modifier;
    int modifier_mask;
} XIMHotKeyTrigger;
typedef struct _XIMHotKeyTriggers {
    int num_hot_key;
    XIMHotKeyTrigger *key;
} XIMHotKeyTriggers;
typedef unsigned long XIMHotKeyState;
typedef struct {
    unsigned short count_values;
    char **supported_values;
} XIMValuesList;
extern int _Xdebug;
extern XFontStruct *XLoadQueryFont(
    Display* ,
    const char*
);
extern XFontStruct *XQueryFont(
    Display* ,
    XID
);
extern XTimeCoord *XGetMotionEvents(
    Display* ,
    Window ,
    Time ,
    Time ,
    int*
);
extern XModifierKeymap *XDeleteModifiermapEntry(
    XModifierKeymap* ,
    KeyCode ,
    int
);
extern XModifierKeymap *XGetModifierMapping(
    Display*
);
extern XModifierKeymap *XInsertModifiermapEntry(
    XModifierKeymap* ,
    KeyCode ,
    int
);
extern XModifierKeymap *XNewModifiermap(
    int
);
extern XImage *XCreateImage(
    Display* ,
    Visual* ,
    unsigned int ,
    int ,
    int ,
    char* ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XInitImage(
    XImage*
);
extern XImage *XGetImage(
    Display* ,
    Drawable ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    unsigned long ,
    int
);
extern XImage *XGetSubImage(
    Display* ,
    Drawable ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    unsigned long ,
    int ,
    XImage* ,
    int ,
    int
);
extern Display *XOpenDisplay(
    const char*
);
extern void XrmInitialize(
    void
);
extern char *XFetchBytes(
    Display* ,
    int*
);
extern char *XFetchBuffer(
    Display* ,
    int* ,
    int
);
extern char *XGetAtomName(
    Display* ,
    Atom
);
extern int XGetAtomNames(
    Display* ,
    Atom* ,
    int ,
    char**
);
extern char *XGetDefault(
    Display* ,
    const char* ,
    const char*
);
extern char *XDisplayName(
    const char*
);
extern char *XKeysymToString(
    KeySym
);
extern int (*XSynchronize(
    Display* ,
    int
))(
    Display*
);
extern int (*XSetAfterFunction(
    Display* ,
    int (*) (
      Display*
            )
))(
    Display*
);
extern Atom XInternAtom(
    Display* ,
    const char* ,
    int
);
extern int XInternAtoms(
    Display* ,
    char** ,
    int ,
    int ,
    Atom*
);
extern Colormap XCopyColormapAndFree(
    Display* ,
    Colormap
);
extern Colormap XCreateColormap(
    Display* ,
    Window ,
    Visual* ,
    int
);
extern Cursor XCreatePixmapCursor(
    Display* ,
    Pixmap ,
    Pixmap ,
    XColor* ,
    XColor* ,
    unsigned int ,
    unsigned int
);
extern Cursor XCreateGlyphCursor(
    Display* ,
    Font ,
    Font ,
    unsigned int ,
    unsigned int ,
    XColor const * ,
    XColor const *
);
extern Cursor XCreateFontCursor(
    Display* ,
    unsigned int
);
extern Font XLoadFont(
    Display* ,
    const char*
);
extern GC XCreateGC(
    Display* ,
    Drawable ,
    unsigned long ,
    XGCValues*
);
extern GContext XGContextFromGC(
    GC
);
extern void XFlushGC(
    Display* ,
    GC
);
extern Pixmap XCreatePixmap(
    Display* ,
    Drawable ,
    unsigned int ,
    unsigned int ,
    unsigned int
);
extern Pixmap XCreateBitmapFromData(
    Display* ,
    Drawable ,
    const char* ,
    unsigned int ,
    unsigned int
);
extern Pixmap XCreatePixmapFromBitmapData(
    Display* ,
    Drawable ,
    char* ,
    unsigned int ,
    unsigned int ,
    unsigned long ,
    unsigned long ,
    unsigned int
);
extern Window XCreateSimpleWindow(
    Display* ,
    Window ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    unsigned int ,
    unsigned long ,
    unsigned long
);
extern Window XGetSelectionOwner(
    Display* ,
    Atom
);
extern Window XCreateWindow(
    Display* ,
    Window ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    unsigned int ,
    int ,
    unsigned int ,
    Visual* ,
    unsigned long ,
    XSetWindowAttributes*
);
extern Colormap *XListInstalledColormaps(
    Display* ,
    Window ,
    int*
);
extern char **XListFonts(
    Display* ,
    const char* ,
    int ,
    int*
);
extern char **XListFontsWithInfo(
    Display* ,
    const char* ,
    int ,
    int* ,
    XFontStruct**
);
extern char **XGetFontPath(
    Display* ,
    int*
);
extern char **XListExtensions(
    Display* ,
    int*
);
extern Atom *XListProperties(
    Display* ,
    Window ,
    int*
);
extern XHostAddress *XListHosts(
    Display* ,
    int* ,
    int*
);
__attribute__((deprecated))
extern KeySym XKeycodeToKeysym(
    Display* ,
    KeyCode ,
    int
);
extern KeySym XLookupKeysym(
    XKeyEvent* ,
    int
);
extern KeySym *XGetKeyboardMapping(
    Display* ,
    KeyCode ,
    int ,
    int*
);
extern KeySym XStringToKeysym(
    const char*
);
extern long XMaxRequestSize(
    Display*
);
extern long XExtendedMaxRequestSize(
    Display*
);
extern char *XResourceManagerString(
    Display*
);
extern char *XScreenResourceString(
 Screen*
);
extern unsigned long XDisplayMotionBufferSize(
    Display*
);
extern VisualID XVisualIDFromVisual(
    Visual*
);
extern int XInitThreads(
    void
);
extern void XLockDisplay(
    Display*
);
extern void XUnlockDisplay(
    Display*
);
extern XExtCodes *XInitExtension(
    Display* ,
    const char*
);
extern XExtCodes *XAddExtension(
    Display*
);
extern XExtData *XFindOnExtensionList(
    XExtData** ,
    int
);
extern XExtData **XEHeadOfExtensionList(
    XEDataObject
);
extern Window XRootWindow(
    Display* ,
    int
);
extern Window XDefaultRootWindow(
    Display*
);
extern Window XRootWindowOfScreen(
    Screen*
);
extern Visual *XDefaultVisual(
    Display* ,
    int
);
extern Visual *XDefaultVisualOfScreen(
    Screen*
);
extern GC XDefaultGC(
    Display* ,
    int
);
extern GC XDefaultGCOfScreen(
    Screen*
);
extern unsigned long XBlackPixel(
    Display* ,
    int
);
extern unsigned long XWhitePixel(
    Display* ,
    int
);
extern unsigned long XAllPlanes(
    void
);
extern unsigned long XBlackPixelOfScreen(
    Screen*
);
extern unsigned long XWhitePixelOfScreen(
    Screen*
);
extern unsigned long XNextRequest(
    Display*
);
extern unsigned long XLastKnownRequestProcessed(
    Display*
);
extern char *XServerVendor(
    Display*
);
extern char *XDisplayString(
    Display*
);
extern Colormap XDefaultColormap(
    Display* ,
    int
);
extern Colormap XDefaultColormapOfScreen(
    Screen*
);
extern Display *XDisplayOfScreen(
    Screen*
);
extern Screen *XScreenOfDisplay(
    Display* ,
    int
);
extern Screen *XDefaultScreenOfDisplay(
    Display*
);
extern long XEventMaskOfScreen(
    Screen*
);
extern int XScreenNumberOfScreen(
    Screen*
);
typedef int (*XErrorHandler) (
    Display* ,
    XErrorEvent*
);
extern XErrorHandler XSetErrorHandler (
    XErrorHandler
);
typedef int (*XIOErrorHandler) (
    Display*
);
extern XIOErrorHandler XSetIOErrorHandler (
    XIOErrorHandler
);
extern XPixmapFormatValues *XListPixmapFormats(
    Display* ,
    int*
);
extern int *XListDepths(
    Display* ,
    int ,
    int*
);
extern int XReconfigureWMWindow(
    Display* ,
    Window ,
    int ,
    unsigned int ,
    XWindowChanges*
);
extern int XGetWMProtocols(
    Display* ,
    Window ,
    Atom** ,
    int*
);
extern int XSetWMProtocols(
    Display* ,
    Window ,
    Atom* ,
    int
);
extern int XIconifyWindow(
    Display* ,
    Window ,
    int
);
extern int XWithdrawWindow(
    Display* ,
    Window ,
    int
);
extern int XGetCommand(
    Display* ,
    Window ,
    char*** ,
    int*
);
extern int XGetWMColormapWindows(
    Display* ,
    Window ,
    Window** ,
    int*
);
extern int XSetWMColormapWindows(
    Display* ,
    Window ,
    Window* ,
    int
);
extern void XFreeStringList(
    char**
);
extern int XSetTransientForHint(
    Display* ,
    Window ,
    Window
);
extern int XActivateScreenSaver(
    Display*
);
extern int XAddHost(
    Display* ,
    XHostAddress*
);
extern int XAddHosts(
    Display* ,
    XHostAddress* ,
    int
);
extern int XAddToExtensionList(
    struct _XExtData** ,
    XExtData*
);
extern int XAddToSaveSet(
    Display* ,
    Window
);
extern int XAllocColor(
    Display* ,
    Colormap ,
    XColor*
);
extern int XAllocColorCells(
    Display* ,
    Colormap ,
    int ,
    unsigned long* ,
    unsigned int ,
    unsigned long* ,
    unsigned int
);
extern int XAllocColorPlanes(
    Display* ,
    Colormap ,
    int ,
    unsigned long* ,
    int ,
    int ,
    int ,
    int ,
    unsigned long* ,
    unsigned long* ,
    unsigned long*
);
extern int XAllocNamedColor(
    Display* ,
    Colormap ,
    const char* ,
    XColor* ,
    XColor*
);
extern int XAllowEvents(
    Display* ,
    int ,
    Time
);
extern int XAutoRepeatOff(
    Display*
);
extern int XAutoRepeatOn(
    Display*
);
extern int XBell(
    Display* ,
    int
);
extern int XBitmapBitOrder(
    Display*
);
extern int XBitmapPad(
    Display*
);
extern int XBitmapUnit(
    Display*
);
extern int XCellsOfScreen(
    Screen*
);
extern int XChangeActivePointerGrab(
    Display* ,
    unsigned int ,
    Cursor ,
    Time
);
extern int XChangeGC(
    Display* ,
    GC ,
    unsigned long ,
    XGCValues*
);
extern int XChangeKeyboardControl(
    Display* ,
    unsigned long ,
    XKeyboardControl*
);
extern int XChangeKeyboardMapping(
    Display* ,
    int ,
    int ,
    KeySym* ,
    int
);
extern int XChangePointerControl(
    Display* ,
    int ,
    int ,
    int ,
    int ,
    int
);
extern int XChangeProperty(
    Display* ,
    Window ,
    Atom ,
    Atom ,
    int ,
    int ,
    const unsigned char* ,
    int
);
extern int XChangeSaveSet(
    Display* ,
    Window ,
    int
);
extern int XChangeWindowAttributes(
    Display* ,
    Window ,
    unsigned long ,
    XSetWindowAttributes*
);
extern int XCheckIfEvent(
    Display* ,
    XEvent* ,
    int (*) (
        Display* ,
               XEvent* ,
               XPointer
             ) ,
    XPointer
);
extern int XCheckMaskEvent(
    Display* ,
    long ,
    XEvent*
);
extern int XCheckTypedEvent(
    Display* ,
    int ,
    XEvent*
);
extern int XCheckTypedWindowEvent(
    Display* ,
    Window ,
    int ,
    XEvent*
);
extern int XCheckWindowEvent(
    Display* ,
    Window ,
    long ,
    XEvent*
);
extern int XCirculateSubwindows(
    Display* ,
    Window ,
    int
);
extern int XCirculateSubwindowsDown(
    Display* ,
    Window
);
extern int XCirculateSubwindowsUp(
    Display* ,
    Window
);
extern int XClearArea(
    Display* ,
    Window ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int
);
extern int XClearWindow(
    Display* ,
    Window
);
extern int XCloseDisplay(
    Display*
);
extern int XConfigureWindow(
    Display* ,
    Window ,
    unsigned int ,
    XWindowChanges*
);
extern int XConnectionNumber(
    Display*
);
extern int XConvertSelection(
    Display* ,
    Atom ,
    Atom ,
    Atom ,
    Window ,
    Time
);
extern int XCopyArea(
    Display* ,
    Drawable ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XCopyGC(
    Display* ,
    GC ,
    unsigned long ,
    GC
);
extern int XCopyPlane(
    Display* ,
    Drawable ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int ,
    int ,
    unsigned long
);
extern int XDefaultDepth(
    Display* ,
    int
);
extern int XDefaultDepthOfScreen(
    Screen*
);
extern int XDefaultScreen(
    Display*
);
extern int XDefineCursor(
    Display* ,
    Window ,
    Cursor
);
extern int XDeleteProperty(
    Display* ,
    Window ,
    Atom
);
extern int XDestroyWindow(
    Display* ,
    Window
);
extern int XDestroySubwindows(
    Display* ,
    Window
);
extern int XDoesBackingStore(
    Screen*
);
extern int XDoesSaveUnders(
    Screen*
);
extern int XDisableAccessControl(
    Display*
);
extern int XDisplayCells(
    Display* ,
    int
);
extern int XDisplayHeight(
    Display* ,
    int
);
extern int XDisplayHeightMM(
    Display* ,
    int
);
extern int XDisplayKeycodes(
    Display* ,
    int* ,
    int*
);
extern int XDisplayPlanes(
    Display* ,
    int
);
extern int XDisplayWidth(
    Display* ,
    int
);
extern int XDisplayWidthMM(
    Display* ,
    int
);
extern int XDrawArc(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XDrawArcs(
    Display* ,
    Drawable ,
    GC ,
    XArc* ,
    int
);
extern int XDrawImageString(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern int XDrawImageString16(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    const XChar2b* ,
    int
);
extern int XDrawLine(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    int ,
    int
);
extern int XDrawLines(
    Display* ,
    Drawable ,
    GC ,
    XPoint* ,
    int ,
    int
);
extern int XDrawPoint(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int
);
extern int XDrawPoints(
    Display* ,
    Drawable ,
    GC ,
    XPoint* ,
    int ,
    int
);
extern int XDrawRectangle(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int
);
extern int XDrawRectangles(
    Display* ,
    Drawable ,
    GC ,
    XRectangle* ,
    int
);
extern int XDrawSegments(
    Display* ,
    Drawable ,
    GC ,
    XSegment* ,
    int
);
extern int XDrawString(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern int XDrawString16(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    const XChar2b* ,
    int
);
extern int XDrawText(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    XTextItem* ,
    int
);
extern int XDrawText16(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    XTextItem16* ,
    int
);
extern int XEnableAccessControl(
    Display*
);
extern int XEventsQueued(
    Display* ,
    int
);
extern int XFetchName(
    Display* ,
    Window ,
    char**
);
extern int XFillArc(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XFillArcs(
    Display* ,
    Drawable ,
    GC ,
    XArc* ,
    int
);
extern int XFillPolygon(
    Display* ,
    Drawable ,
    GC ,
    XPoint* ,
    int ,
    int ,
    int
);
extern int XFillRectangle(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    unsigned int ,
    unsigned int
);
extern int XFillRectangles(
    Display* ,
    Drawable ,
    GC ,
    XRectangle* ,
    int
);
extern int XFlush(
    Display*
);
extern int XForceScreenSaver(
    Display* ,
    int
);
extern int XFree(
    void*
);
extern int XFreeColormap(
    Display* ,
    Colormap
);
extern int XFreeColors(
    Display* ,
    Colormap ,
    unsigned long* ,
    int ,
    unsigned long
);
extern int XFreeCursor(
    Display* ,
    Cursor
);
extern int XFreeExtensionList(
    char**
);
extern int XFreeFont(
    Display* ,
    XFontStruct*
);
extern int XFreeFontInfo(
    char** ,
    XFontStruct* ,
    int
);
extern int XFreeFontNames(
    char**
);
extern int XFreeFontPath(
    char**
);
extern int XFreeGC(
    Display* ,
    GC
);
extern int XFreeModifiermap(
    XModifierKeymap*
);
extern int XFreePixmap(
    Display* ,
    Pixmap
);
extern int XGeometry(
    Display* ,
    int ,
    const char* ,
    const char* ,
    unsigned int ,
    unsigned int ,
    unsigned int ,
    int ,
    int ,
    int* ,
    int* ,
    int* ,
    int*
);
extern int XGetErrorDatabaseText(
    Display* ,
    const char* ,
    const char* ,
    const char* ,
    char* ,
    int
);
extern int XGetErrorText(
    Display* ,
    int ,
    char* ,
    int
);
extern int XGetFontProperty(
    XFontStruct* ,
    Atom ,
    unsigned long*
);
extern int XGetGCValues(
    Display* ,
    GC ,
    unsigned long ,
    XGCValues*
);
extern int XGetGeometry(
    Display* ,
    Drawable ,
    Window* ,
    int* ,
    int* ,
    unsigned int* ,
    unsigned int* ,
    unsigned int* ,
    unsigned int*
);
extern int XGetIconName(
    Display* ,
    Window ,
    char**
);
extern int XGetInputFocus(
    Display* ,
    Window* ,
    int*
);
extern int XGetKeyboardControl(
    Display* ,
    XKeyboardState*
);
extern int XGetPointerControl(
    Display* ,
    int* ,
    int* ,
    int*
);
extern int XGetPointerMapping(
    Display* ,
    unsigned char* ,
    int
);
extern int XGetScreenSaver(
    Display* ,
    int* ,
    int* ,
    int* ,
    int*
);
extern int XGetTransientForHint(
    Display* ,
    Window ,
    Window*
);
extern int XGetWindowProperty(
    Display* ,
    Window ,
    Atom ,
    long ,
    long ,
    int ,
    Atom ,
    Atom* ,
    int* ,
    unsigned long* ,
    unsigned long* ,
    unsigned char**
);
extern int XGetWindowAttributes(
    Display* ,
    Window ,
    XWindowAttributes*
);
extern int XGrabButton(
    Display* ,
    unsigned int ,
    unsigned int ,
    Window ,
    int ,
    unsigned int ,
    int ,
    int ,
    Window ,
    Cursor
);
extern int XGrabKey(
    Display* ,
    int ,
    unsigned int ,
    Window ,
    int ,
    int ,
    int
);
extern int XGrabKeyboard(
    Display* ,
    Window ,
    int ,
    int ,
    int ,
    Time
);
extern int XGrabPointer(
    Display* ,
    Window ,
    int ,
    unsigned int ,
    int ,
    int ,
    Window ,
    Cursor ,
    Time
);
extern int XGrabServer(
    Display*
);
extern int XHeightMMOfScreen(
    Screen*
);
extern int XHeightOfScreen(
    Screen*
);
extern int XIfEvent(
    Display* ,
    XEvent* ,
    int (*) (
        Display* ,
               XEvent* ,
               XPointer
             ) ,
    XPointer
);
extern int XImageByteOrder(
    Display*
);
extern int XInstallColormap(
    Display* ,
    Colormap
);
extern KeyCode XKeysymToKeycode(
    Display* ,
    KeySym
);
extern int XKillClient(
    Display* ,
    XID
);
extern int XLookupColor(
    Display* ,
    Colormap ,
    const char* ,
    XColor* ,
    XColor*
);
extern int XLowerWindow(
    Display* ,
    Window
);
extern int XMapRaised(
    Display* ,
    Window
);
extern int XMapSubwindows(
    Display* ,
    Window
);
extern int XMapWindow(
    Display* ,
    Window
);
extern int XMaskEvent(
    Display* ,
    long ,
    XEvent*
);
extern int XMaxCmapsOfScreen(
    Screen*
);
extern int XMinCmapsOfScreen(
    Screen*
);
extern int XMoveResizeWindow(
    Display* ,
    Window ,
    int ,
    int ,
    unsigned int ,
    unsigned int
);
extern int XMoveWindow(
    Display* ,
    Window ,
    int ,
    int
);
extern int XNextEvent(
    Display* ,
    XEvent*
);
extern int XNoOp(
    Display*
);
extern int XParseColor(
    Display* ,
    Colormap ,
    const char* ,
    XColor*
);
extern int XParseGeometry(
    const char* ,
    int* ,
    int* ,
    unsigned int* ,
    unsigned int*
);
extern int XPeekEvent(
    Display* ,
    XEvent*
);
extern int XPeekIfEvent(
    Display* ,
    XEvent* ,
    int (*) (
        Display* ,
               XEvent* ,
               XPointer
             ) ,
    XPointer
);
extern int XPending(
    Display*
);
extern int XPlanesOfScreen(
    Screen*
);
extern int XProtocolRevision(
    Display*
);
extern int XProtocolVersion(
    Display*
);
extern int XPutBackEvent(
    Display* ,
    XEvent*
);
extern int XPutImage(
    Display* ,
    Drawable ,
    GC ,
    XImage* ,
    int ,
    int ,
    int ,
    int ,
    unsigned int ,
    unsigned int
);
extern int XQLength(
    Display*
);
extern int XQueryBestCursor(
    Display* ,
    Drawable ,
    unsigned int ,
    unsigned int ,
    unsigned int* ,
    unsigned int*
);
extern int XQueryBestSize(
    Display* ,
    int ,
    Drawable ,
    unsigned int ,
    unsigned int ,
    unsigned int* ,
    unsigned int*
);
extern int XQueryBestStipple(
    Display* ,
    Drawable ,
    unsigned int ,
    unsigned int ,
    unsigned int* ,
    unsigned int*
);
extern int XQueryBestTile(
    Display* ,
    Drawable ,
    unsigned int ,
    unsigned int ,
    unsigned int* ,
    unsigned int*
);
extern int XQueryColor(
    Display* ,
    Colormap ,
    XColor*
);
extern int XQueryColors(
    Display* ,
    Colormap ,
    XColor* ,
    int
);
extern int XQueryExtension(
    Display* ,
    const char* ,
    int* ,
    int* ,
    int*
);
extern int XQueryKeymap(
    Display* ,
    char [32]
);
extern int XQueryPointer(
    Display* ,
    Window ,
    Window* ,
    Window* ,
    int* ,
    int* ,
    int* ,
    int* ,
    unsigned int*
);
extern int XQueryTextExtents(
    Display* ,
    XID ,
    const char* ,
    int ,
    int* ,
    int* ,
    int* ,
    XCharStruct*
);
extern int XQueryTextExtents16(
    Display* ,
    XID ,
    const XChar2b* ,
    int ,
    int* ,
    int* ,
    int* ,
    XCharStruct*
);
extern int XQueryTree(
    Display* ,
    Window ,
    Window* ,
    Window* ,
    Window** ,
    unsigned int*
);
extern int XRaiseWindow(
    Display* ,
    Window
);
extern int XReadBitmapFile(
    Display* ,
    Drawable ,
    const char* ,
    unsigned int* ,
    unsigned int* ,
    Pixmap* ,
    int* ,
    int*
);
extern int XReadBitmapFileData(
    const char* ,
    unsigned int* ,
    unsigned int* ,
    unsigned char** ,
    int* ,
    int*
);
extern int XRebindKeysym(
    Display* ,
    KeySym ,
    KeySym* ,
    int ,
    const unsigned char* ,
    int
);
extern int XRecolorCursor(
    Display* ,
    Cursor ,
    XColor* ,
    XColor*
);
extern int XRefreshKeyboardMapping(
    XMappingEvent*
);
extern int XRemoveFromSaveSet(
    Display* ,
    Window
);
extern int XRemoveHost(
    Display* ,
    XHostAddress*
);
extern int XRemoveHosts(
    Display* ,
    XHostAddress* ,
    int
);
extern int XReparentWindow(
    Display* ,
    Window ,
    Window ,
    int ,
    int
);
extern int XResetScreenSaver(
    Display*
);
extern int XResizeWindow(
    Display* ,
    Window ,
    unsigned int ,
    unsigned int
);
extern int XRestackWindows(
    Display* ,
    Window* ,
    int
);
extern int XRotateBuffers(
    Display* ,
    int
);
extern int XRotateWindowProperties(
    Display* ,
    Window ,
    Atom* ,
    int ,
    int
);
extern int XScreenCount(
    Display*
);
extern int XSelectInput(
    Display* ,
    Window ,
    long
);
extern int XSendEvent(
    Display* ,
    Window ,
    int ,
    long ,
    XEvent*
);
extern int XSetAccessControl(
    Display* ,
    int
);
extern int XSetArcMode(
    Display* ,
    GC ,
    int
);
extern int XSetBackground(
    Display* ,
    GC ,
    unsigned long
);
extern int XSetClipMask(
    Display* ,
    GC ,
    Pixmap
);
extern int XSetClipOrigin(
    Display* ,
    GC ,
    int ,
    int
);
extern int XSetClipRectangles(
    Display* ,
    GC ,
    int ,
    int ,
    XRectangle* ,
    int ,
    int
);
extern int XSetCloseDownMode(
    Display* ,
    int
);
extern int XSetCommand(
    Display* ,
    Window ,
    char** ,
    int
);
extern int XSetDashes(
    Display* ,
    GC ,
    int ,
    const char* ,
    int
);
extern int XSetFillRule(
    Display* ,
    GC ,
    int
);
extern int XSetFillStyle(
    Display* ,
    GC ,
    int
);
extern int XSetFont(
    Display* ,
    GC ,
    Font
);
extern int XSetFontPath(
    Display* ,
    char** ,
    int
);
extern int XSetForeground(
    Display* ,
    GC ,
    unsigned long
);
extern int XSetFunction(
    Display* ,
    GC ,
    int
);
extern int XSetGraphicsExposures(
    Display* ,
    GC ,
    int
);
extern int XSetIconName(
    Display* ,
    Window ,
    const char*
);
extern int XSetInputFocus(
    Display* ,
    Window ,
    int ,
    Time
);
extern int XSetLineAttributes(
    Display* ,
    GC ,
    unsigned int ,
    int ,
    int ,
    int
);
extern int XSetModifierMapping(
    Display* ,
    XModifierKeymap*
);
extern int XSetPlaneMask(
    Display* ,
    GC ,
    unsigned long
);
extern int XSetPointerMapping(
    Display* ,
    const unsigned char* ,
    int
);
extern int XSetScreenSaver(
    Display* ,
    int ,
    int ,
    int ,
    int
);
extern int XSetSelectionOwner(
    Display* ,
    Atom ,
    Window ,
    Time
);
extern int XSetState(
    Display* ,
    GC ,
    unsigned long ,
    unsigned long ,
    int ,
    unsigned long
);
extern int XSetStipple(
    Display* ,
    GC ,
    Pixmap
);
extern int XSetSubwindowMode(
    Display* ,
    GC ,
    int
);
extern int XSetTSOrigin(
    Display* ,
    GC ,
    int ,
    int
);
extern int XSetTile(
    Display* ,
    GC ,
    Pixmap
);
extern int XSetWindowBackground(
    Display* ,
    Window ,
    unsigned long
);
extern int XSetWindowBackgroundPixmap(
    Display* ,
    Window ,
    Pixmap
);
extern int XSetWindowBorder(
    Display* ,
    Window ,
    unsigned long
);
extern int XSetWindowBorderPixmap(
    Display* ,
    Window ,
    Pixmap
);
extern int XSetWindowBorderWidth(
    Display* ,
    Window ,
    unsigned int
);
extern int XSetWindowColormap(
    Display* ,
    Window ,
    Colormap
);
extern int XStoreBuffer(
    Display* ,
    const char* ,
    int ,
    int
);
extern int XStoreBytes(
    Display* ,
    const char* ,
    int
);
extern int XStoreColor(
    Display* ,
    Colormap ,
    XColor*
);
extern int XStoreColors(
    Display* ,
    Colormap ,
    XColor* ,
    int
);
extern int XStoreName(
    Display* ,
    Window ,
    const char*
);
extern int XStoreNamedColor(
    Display* ,
    Colormap ,
    const char* ,
    unsigned long ,
    int
);
extern int XSync(
    Display* ,
    int
);
extern int XTextExtents(
    XFontStruct* ,
    const char* ,
    int ,
    int* ,
    int* ,
    int* ,
    XCharStruct*
);
extern int XTextExtents16(
    XFontStruct* ,
    const XChar2b* ,
    int ,
    int* ,
    int* ,
    int* ,
    XCharStruct*
);
extern int XTextWidth(
    XFontStruct* ,
    const char* ,
    int
);
extern int XTextWidth16(
    XFontStruct* ,
    const XChar2b* ,
    int
);
extern int XTranslateCoordinates(
    Display* ,
    Window ,
    Window ,
    int ,
    int ,
    int* ,
    int* ,
    Window*
);
extern int XUndefineCursor(
    Display* ,
    Window
);
extern int XUngrabButton(
    Display* ,
    unsigned int ,
    unsigned int ,
    Window
);
extern int XUngrabKey(
    Display* ,
    int ,
    unsigned int ,
    Window
);
extern int XUngrabKeyboard(
    Display* ,
    Time
);
extern int XUngrabPointer(
    Display* ,
    Time
);
extern int XUngrabServer(
    Display*
);
extern int XUninstallColormap(
    Display* ,
    Colormap
);
extern int XUnloadFont(
    Display* ,
    Font
);
extern int XUnmapSubwindows(
    Display* ,
    Window
);
extern int XUnmapWindow(
    Display* ,
    Window
);
extern int XVendorRelease(
    Display*
);
extern int XWarpPointer(
    Display* ,
    Window ,
    Window ,
    int ,
    int ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XWidthMMOfScreen(
    Screen*
);
extern int XWidthOfScreen(
    Screen*
);
extern int XWindowEvent(
    Display* ,
    Window ,
    long ,
    XEvent*
);
extern int XWriteBitmapFile(
    Display* ,
    const char* ,
    Pixmap ,
    unsigned int ,
    unsigned int ,
    int ,
    int
);
extern int XSupportsLocale (void);
extern char *XSetLocaleModifiers(
    const char*
);
extern XOM XOpenOM(
    Display* ,
    struct _XrmHashBucketRec* ,
    const char* ,
    const char*
);
extern int XCloseOM(
    XOM
);
extern char *XSetOMValues(
    XOM ,
    ...
) __attribute__ ((__sentinel__(0)));
extern char *XGetOMValues(
    XOM ,
    ...
) __attribute__ ((__sentinel__(0)));
extern Display *XDisplayOfOM(
    XOM
);
extern char *XLocaleOfOM(
    XOM
);
extern XOC XCreateOC(
    XOM ,
    ...
) __attribute__ ((__sentinel__(0)));
extern void XDestroyOC(
    XOC
);
extern XOM XOMOfOC(
    XOC
);
extern char *XSetOCValues(
    XOC ,
    ...
) __attribute__ ((__sentinel__(0)));
extern char *XGetOCValues(
    XOC ,
    ...
) __attribute__ ((__sentinel__(0)));
extern XFontSet XCreateFontSet(
    Display* ,
    const char* ,
    char*** ,
    int* ,
    char**
);
extern void XFreeFontSet(
    Display* ,
    XFontSet
);
extern int XFontsOfFontSet(
    XFontSet ,
    XFontStruct*** ,
    char***
);
extern char *XBaseFontNameListOfFontSet(
    XFontSet
);
extern char *XLocaleOfFontSet(
    XFontSet
);
extern int XContextDependentDrawing(
    XFontSet
);
extern int XDirectionalDependentDrawing(
    XFontSet
);
extern int XContextualDrawing(
    XFontSet
);
extern XFontSetExtents *XExtentsOfFontSet(
    XFontSet
);
extern int XmbTextEscapement(
    XFontSet ,
    const char* ,
    int
);
extern int XwcTextEscapement(
    XFontSet ,
    const wchar_t* ,
    int
);
extern int Xutf8TextEscapement(
    XFontSet ,
    const char* ,
    int
);
extern int XmbTextExtents(
    XFontSet ,
    const char* ,
    int ,
    XRectangle* ,
    XRectangle*
);
extern int XwcTextExtents(
    XFontSet ,
    const wchar_t* ,
    int ,
    XRectangle* ,
    XRectangle*
);
extern int Xutf8TextExtents(
    XFontSet ,
    const char* ,
    int ,
    XRectangle* ,
    XRectangle*
);
extern int XmbTextPerCharExtents(
    XFontSet ,
    const char* ,
    int ,
    XRectangle* ,
    XRectangle* ,
    int ,
    int* ,
    XRectangle* ,
    XRectangle*
);
extern int XwcTextPerCharExtents(
    XFontSet ,
    const wchar_t* ,
    int ,
    XRectangle* ,
    XRectangle* ,
    int ,
    int* ,
    XRectangle* ,
    XRectangle*
);
extern int Xutf8TextPerCharExtents(
    XFontSet ,
    const char* ,
    int ,
    XRectangle* ,
    XRectangle* ,
    int ,
    int* ,
    XRectangle* ,
    XRectangle*
);
extern void XmbDrawText(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    XmbTextItem* ,
    int
);
extern void XwcDrawText(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    XwcTextItem* ,
    int
);
extern void Xutf8DrawText(
    Display* ,
    Drawable ,
    GC ,
    int ,
    int ,
    XmbTextItem* ,
    int
);
extern void XmbDrawString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern void XwcDrawString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const wchar_t* ,
    int
);
extern void Xutf8DrawString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern void XmbDrawImageString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern void XwcDrawImageString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const wchar_t* ,
    int
);
extern void Xutf8DrawImageString(
    Display* ,
    Drawable ,
    XFontSet ,
    GC ,
    int ,
    int ,
    const char* ,
    int
);
extern XIM XOpenIM(
    Display* ,
    struct _XrmHashBucketRec* ,
    char* ,
    char*
);
extern int XCloseIM(
    XIM
);
extern char *XGetIMValues(
    XIM , ...
) __attribute__ ((__sentinel__(0)));
extern char *XSetIMValues(
    XIM , ...
) __attribute__ ((__sentinel__(0)));
extern Display *XDisplayOfIM(
    XIM
);
extern char *XLocaleOfIM(
    XIM
);
extern XIC XCreateIC(
    XIM , ...
) __attribute__ ((__sentinel__(0)));
extern void XDestroyIC(
    XIC
);
extern void XSetICFocus(
    XIC
);
extern void XUnsetICFocus(
    XIC
);
extern wchar_t *XwcResetIC(
    XIC
);
extern char *XmbResetIC(
    XIC
);
extern char *Xutf8ResetIC(
    XIC
);
extern char *XSetICValues(
    XIC , ...
) __attribute__ ((__sentinel__(0)));
extern char *XGetICValues(
    XIC , ...
) __attribute__ ((__sentinel__(0)));
extern XIM XIMOfIC(
    XIC
);
extern int XFilterEvent(
    XEvent* ,
    Window
);
extern int XmbLookupString(
    XIC ,
    XKeyPressedEvent* ,
    char* ,
    int ,
    KeySym* ,
    int*
);
extern int XwcLookupString(
    XIC ,
    XKeyPressedEvent* ,
    wchar_t* ,
    int ,
    KeySym* ,
    int*
);
extern int Xutf8LookupString(
    XIC ,
    XKeyPressedEvent* ,
    char* ,
    int ,
    KeySym* ,
    int*
);
extern XVaNestedList XVaCreateNestedList(
    int , ...
) __attribute__ ((__sentinel__(0)));
extern int XRegisterIMInstantiateCallback(
    Display* ,
    struct _XrmHashBucketRec* ,
    char* ,
    char* ,
    XIDProc ,
    XPointer
);
extern int XUnregisterIMInstantiateCallback(
    Display* ,
    struct _XrmHashBucketRec* ,
    char* ,
    char* ,
    XIDProc ,
    XPointer
);
typedef void (*XConnectionWatchProc)(
    Display* ,
    XPointer ,
    int ,
    int ,
    XPointer*
);
extern int XInternalConnectionNumbers(
    Display* ,
    int** ,
    int*
);
extern void XProcessInternalConnection(
    Display* ,
    int
);
extern int XAddConnectionWatch(
    Display* ,
    XConnectionWatchProc ,
    XPointer
);
extern void XRemoveConnectionWatch(
    Display* ,
    XConnectionWatchProc ,
    XPointer
);
extern void XSetAuthorization(
    char * ,
    int ,
    char * ,
    int
);
extern int _Xmbtowc(
    wchar_t * ,
    char * ,
    int
);
extern int _Xwctomb(
    char * ,
    wchar_t
);
extern int XGetEventData(
    Display* ,
    XGenericEventCookie*
);
extern void XFreeEventData(
    Display* ,
    XGenericEventCookie*
);
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['X11/Xlib.h'] in ['/usr/include', '/usr/lib/openmpi']
                Popping language C
================================================================================
TEST checkSharedLibrary from config.packages.X(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.X(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.silo(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.silo(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.Random123(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Random123(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.PARTY(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.PARTY(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.Numpy(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Numpy(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.petsc4py(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/petsc4py.py:106)
TESTING: alternateConfigureLibrary from config.packages.petsc4py(config/BuildSystem/config/packages/petsc4py.py:106)
              Defined make rule "petsc4py-build" with dependencies "" and code []
              Defined make rule "petsc4py-install" with dependencies "" and code []
================================================================================
TEST alternateConfigureLibrary from config.packages.mpi4py(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/mpi4py.py:71)
TESTING: alternateConfigureLibrary from config.packages.mpi4py(config/BuildSystem/config/packages/mpi4py.py:71)
              Defined make rule "mpi4py-build" with dependencies "" and code []
              Defined make rule "mpi4py-install" with dependencies "" and code []
================================================================================
TEST alternateConfigureLibrary from config.packages.Matlab(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Matlab(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.MatlabEngine(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.MatlabEngine(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.Mathematica(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Mathematica(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.CoDiPack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.CoDiPack(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.adblaslapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.adblaslapack(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.szlib(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.szlib(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.zlib(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.zlib(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST locateCMake from config.packages.cmake(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/cmake.py:36)
TESTING: locateCMake from config.packages.cmake(config/BuildSystem/config/packages/cmake.py:36)
Looking for default CMake executable
Checking for program /home/ryan/bin/cmake...not found
Checking for program /usr/local/bin/cmake...not found
Checking for program /home/ryan/bin/cmake...not found
Checking for program /usr/local/bin/cmake...not found
Checking for program /usr/local/sbin/cmake...not found
Checking for program /usr/bin/cmake...found
                Defined make macro "CMAKE" to "/usr/bin/cmake"
Looking for default CTest executable
Checking for program /home/ryan/bin/ctest...not found
Checking for program /usr/local/bin/ctest...not found
Checking for program /home/ryan/bin/ctest...not found
Checking for program /usr/local/bin/ctest...not found
Checking for program /usr/local/sbin/ctest...not found
Checking for program /usr/bin/ctest...found
                Defined make macro "CTEST" to "/usr/bin/ctest"
================================================================================
TEST alternateConfigureLibrary from config.packages.googletest(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.googletest(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.unittestcpp(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.unittestcpp(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.eigen(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.eigen(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.tetgen(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.tetgen(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.tchem(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.tchem(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.saws(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.saws(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.libpng(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.libpng(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.combblas(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.combblas(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.Triangle(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Triangle(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.PTScotch(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.PTScotch(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.metis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.metis(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.metis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/metis.py:39)
TESTING: configureLibrary from config.packages.metis(config/BuildSystem/config/packages/metis.py:39)
==================================================================================
              Checking for a functional metis
                      Looking for METIS at git.metis, hg.metis or a directory starting with ['petsc-pkg-metis']
                      Found a copy of METIS in git.metis
Executing: ['git', 'rev-parse', '--git-dir']
stdout: .git
Executing: ['git', 'cat-file', '-e', 'v5.1.0-p5^{commit}']
Executing: ['git', 'rev-parse', 'v5.1.0-p5']
stdout: 49e61501c498f7a70bde4171fe99aee9566f1a28
Executing: ['git', 'stash']
stdout: No local changes to save
Executing: ['git', 'clean', '-f', '-d', '-x']
stdout:
Removing metis.petscconf
Removing pkg.gitcommit
Executing: ['git', 'checkout', '-f', '49e61501c498f7a70bde4171fe99aee9566f1a28']
                            Pushing language C
                            Popping language C
                            Pushing language Cxx
                            Popping language Cxx
                            Pushing language FC
                            Popping language FC
Have to rebuild METIS, /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/metis.petscconf != /home/ryan/projects/moose/petsc/linux-c-opt/lib/petsc/conf/pkg.conf.metis
                          ===============================================================================
                              Configuring METIS with cmake, this may take several minutes
                          ===============================================================================
                        
Executing: cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/home/ryan/projects/moose/petsc/linux-c-opt -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_COMPILER="mpicxx" -DCMAKE_CXX_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_COMPILER="mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DBUILD_SHARED_LIBS=on -DGKLIB_PATH=../GKlib -DGKRAND=1 -DSHARED=1 -DMATH_LIB="-lm"
stdout:
-- The C compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for execinfo.h
-- Looking for execinfo.h - found
-- Looking for getline
-- Looking for getline - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build
                          ===============================================================================
                              Compiling and installing METIS; this may take several minutes
                          ===============================================================================
                        
Executing: cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/make -j4 -l6.0  &&  /usr/bin/make install
stdout:
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis/CMakeFiles/metis.dir/DependInfo.cmake --color=
Scanning dependencies of target metis
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[  1%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/b64.c.o
[  3%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/blas.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/blas.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/blas.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/b64.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/b64.c
[  4%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/csr.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/csr.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c
[  6%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/error.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/error.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/error.c
[  8%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/evaluate.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/evaluate.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/evaluate.c
[  9%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/fkvkselect.c
[ 11%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/fs.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/fs.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/fs.c
[ 12%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/getopt.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/getopt.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c
[ 14%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/gkregex.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/gkregex.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gkregex.c
[ 16%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/graph.c
[ 17%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/htable.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/htable.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/htable.c
[ 19%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/io.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/io.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/io.c
[ 20%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/itemsets.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/itemsets.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/itemsets.c
[ 22%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/mcore.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/mcore.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/mcore.c
[ 24%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/memory.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/memory.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/memory.c
[ 25%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/omp.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/omp.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/omp.c
[ 27%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/pdb.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/pdb.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pdb.c
[ 29%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/pqueue.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/pqueue.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c
[ 30%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/random.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/random.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/random.c
[ 32%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/rw.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/rw.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/rw.c
[ 33%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/seq.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/seq.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/seq.c
[ 35%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/sort.c.o
[ 37%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/string.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/string.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/string.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/sort.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/sort.c
[ 38%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/timers.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/timers.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/timers.c
[ 40%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/tokenizer.c
[ 41%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/util.c
[ 43%] Building C object libmetis/CMakeFiles/metis.dir/auxapi.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/auxapi.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/auxapi.c
[ 45%] Building C object libmetis/CMakeFiles/metis.dir/balance.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/balance.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/balance.c
[ 46%] Building C object libmetis/CMakeFiles/metis.dir/bucketsort.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/bucketsort.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/bucketsort.c
[ 48%] Building C object libmetis/CMakeFiles/metis.dir/checkgraph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/checkgraph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/checkgraph.c
[ 50%] Building C object libmetis/CMakeFiles/metis.dir/coarsen.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/coarsen.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/coarsen.c
[ 51%] Building C object libmetis/CMakeFiles/metis.dir/compress.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/compress.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/compress.c
[ 53%] Building C object libmetis/CMakeFiles/metis.dir/contig.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/contig.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c
[ 54%] Building C object libmetis/CMakeFiles/metis.dir/debug.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/debug.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c
[ 56%] Building C object libmetis/CMakeFiles/metis.dir/fm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/fm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/fm.c
[ 58%] Building C object libmetis/CMakeFiles/metis.dir/fortran.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/fortran.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/fortran.c
[ 59%] Building C object libmetis/CMakeFiles/metis.dir/frename.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/frename.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/frename.c
[ 61%] Building C object libmetis/CMakeFiles/metis.dir/gklib.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/gklib.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c
[ 62%] Building C object libmetis/CMakeFiles/metis.dir/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/graph.c
[ 64%] Building C object libmetis/CMakeFiles/metis.dir/initpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/initpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c
[ 66%] Building C object libmetis/CMakeFiles/metis.dir/kmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kmetis.c
[ 67%] Building C object libmetis/CMakeFiles/metis.dir/kwayfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kwayfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c
[ 69%] Building C object libmetis/CMakeFiles/metis.dir/kwayrefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kwayrefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayrefine.c
[ 70%] Building C object libmetis/CMakeFiles/metis.dir/mcutil.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mcutil.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mcutil.c
[ 72%] Building C object libmetis/CMakeFiles/metis.dir/mesh.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mesh.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mesh.c
[ 74%] Building C object libmetis/CMakeFiles/metis.dir/meshpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/meshpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/meshpart.c
[ 75%] Building C object libmetis/CMakeFiles/metis.dir/minconn.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/minconn.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c
[ 77%] Building C object libmetis/CMakeFiles/metis.dir/mincover.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mincover.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mincover.c
[ 79%] Building C object libmetis/CMakeFiles/metis.dir/mmd.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mmd.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mmd.c
[ 80%] Building C object libmetis/CMakeFiles/metis.dir/ometis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/ometis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c
[ 82%] Building C object libmetis/CMakeFiles/metis.dir/options.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/options.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/options.c
[ 83%] Building C object libmetis/CMakeFiles/metis.dir/parmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/parmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/parmetis.c
[ 85%] Building C object libmetis/CMakeFiles/metis.dir/pmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/pmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/pmetis.c
[ 87%] Building C object libmetis/CMakeFiles/metis.dir/refine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/refine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/refine.c
[ 88%] Building C object libmetis/CMakeFiles/metis.dir/separator.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/separator.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/separator.c
[ 90%] Building C object libmetis/CMakeFiles/metis.dir/sfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/sfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/sfm.c
[ 91%] Building C object libmetis/CMakeFiles/metis.dir/srefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/srefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/srefine.c
[ 93%] Building C object libmetis/CMakeFiles/metis.dir/stat.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/stat.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/stat.c
[ 95%] Building C object libmetis/CMakeFiles/metis.dir/timing.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/timing.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/timing.c
[ 96%] Building C object libmetis/CMakeFiles/metis.dir/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/util.c
[ 98%] Building C object libmetis/CMakeFiles/metis.dir/wspace.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/wspace.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/wspace.c
[100%] Linking C shared library libmetis.so
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/cmake -E cmake_link_script CMakeFiles/metis.dir/link.txt --verbose=1
/usr/bin/mpicc -fPIC -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE   -shared -Wl,-soname,libmetis.so -o libmetis.so CMakeFiles/metis.dir/__/GKlib/b64.c.o CMakeFiles/metis.dir/__/GKlib/blas.c.o CMakeFiles/metis.dir/__/GKlib/csr.c.o CMakeFiles/metis.dir/__/GKlib/error.c.o CMakeFiles/metis.dir/__/GKlib/evaluate.c.o CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o CMakeFiles/metis.dir/__/GKlib/fs.c.o CMakeFiles/metis.dir/__/GKlib/getopt.c.o CMakeFiles/metis.dir/__/GKlib/gkregex.c.o CMakeFiles/metis.dir/__/GKlib/graph.c.o CMakeFiles/metis.dir/__/GKlib/htable.c.o CMakeFiles/metis.dir/__/GKlib/io.c.o CMakeFiles/metis.dir/__/GKlib/itemsets.c.o CMakeFiles/metis.dir/__/GKlib/mcore.c.o CMakeFiles/metis.dir/__/GKlib/memory.c.o CMakeFiles/metis.dir/__/GKlib/omp.c.o CMakeFiles/metis.dir/__/GKlib/pdb.c.o CMakeFiles/metis.dir/__/GKlib/pqueue.c.o CMakeFiles/metis.dir/__/GKlib/random.c.o CMakeFiles/metis.dir/__/GKlib/rw.c.o CMakeFiles/metis.dir/__/GKlib/seq.c.o CMakeFiles/metis.dir/__/GKlib/sort.c.o CMakeFiles/metis.dir/__/GKlib/string.c.o CMakeFiles/metis.dir/__/GKlib/timers.c.o CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o CMakeFiles/metis.dir/__/GKlib/util.c.o CMakeFiles/metis.dir/auxapi.c.o CMakeFiles/metis.dir/balance.c.o CMakeFiles/metis.dir/bucketsort.c.o CMakeFiles/metis.dir/checkgraph.c.o CMakeFiles/metis.dir/coarsen.c.o CMakeFiles/metis.dir/compress.c.o CMakeFiles/metis.dir/contig.c.o CMakeFiles/metis.dir/debug.c.o CMakeFiles/metis.dir/fm.c.o CMakeFiles/metis.dir/fortran.c.o CMakeFiles/metis.dir/frename.c.o CMakeFiles/metis.dir/gklib.c.o CMakeFiles/metis.dir/graph.c.o CMakeFiles/metis.dir/initpart.c.o CMakeFiles/metis.dir/kmetis.c.o CMakeFiles/metis.dir/kwayfm.c.o CMakeFiles/metis.dir/kwayrefine.c.o CMakeFiles/metis.dir/mcutil.c.o CMakeFiles/metis.dir/mesh.c.o CMakeFiles/metis.dir/meshpart.c.o CMakeFiles/metis.dir/minconn.c.o CMakeFiles/metis.dir/mincover.c.o CMakeFiles/metis.dir/mmd.c.o CMakeFiles/metis.dir/ometis.c.o CMakeFiles/metis.dir/options.c.o CMakeFiles/metis.dir/parmetis.c.o CMakeFiles/metis.dir/pmetis.c.o CMakeFiles/metis.dir/refine.c.o CMakeFiles/metis.dir/separator.c.o CMakeFiles/metis.dir/sfm.c.o CMakeFiles/metis.dir/srefine.c.o CMakeFiles/metis.dir/stat.c.o CMakeFiles/metis.dir/timing.c.o CMakeFiles/metis.dir/util.c.o CMakeFiles/metis.dir/wspace.c.o -lm 
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[100%] Built target metis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles 0
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis/CMakeFiles/metis.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
make[2]: Nothing to be done for 'libmetis/CMakeFiles/metis.dir/build'.
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[100%] Built target metis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles 0
/usr/bin/make -f CMakeFiles/Makefile2 preinstall
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
make[1]: Nothing to be done for 'preinstall'.
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_tls.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.so
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_defs.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_rename.h
********Output of running make on METIS follows *******
-- The C compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for execinfo.h
-- Looking for execinfo.h - found
-- Looking for getline
-- Looking for getline - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-buildCMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS
    CMAKE_CXX_FLAGS_DEBUG
    CMAKE_CXX_FLAGS_RELEASE
    CMAKE_C_FLAGS_DEBUG
    CMAKE_C_FLAGS_RELEASE
    CMAKE_Fortran_COMPILER
    CMAKE_Fortran_FLAGS
    CMAKE_Fortran_FLAGS_DEBUG
    CMAKE_Fortran_FLAGS_RELEASE


/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis/CMakeFiles/metis.dir/DependInfo.cmake --color=
Scanning dependencies of target metis
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[  1%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/b64.c.o
[  3%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/blas.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/blas.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/blas.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/b64.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/b64.c
[  4%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/csr.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/csr.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c
[  6%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/error.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/error.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/error.c
[  8%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/evaluate.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/evaluate.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/evaluate.c
[  9%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/fkvkselect.c
[ 11%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/fs.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/fs.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/fs.c
[ 12%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/getopt.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/getopt.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c
[ 14%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/gkregex.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/gkregex.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gkregex.c
[ 16%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/graph.c
[ 17%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/htable.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/htable.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/htable.c
[ 19%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/io.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/io.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/io.c
[ 20%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/itemsets.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/itemsets.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/itemsets.c
[ 22%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/mcore.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/mcore.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/mcore.c
[ 24%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/memory.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/memory.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/memory.c
[ 25%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/omp.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/omp.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/omp.c
[ 27%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/pdb.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/pdb.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pdb.c
[ 29%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/pqueue.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/pqueue.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c
[ 30%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/random.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/random.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/random.c
[ 32%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/rw.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/rw.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/rw.c
[ 33%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/seq.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/seq.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/seq.c
[ 35%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/sort.c.o
[ 37%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/string.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/string.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/string.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/sort.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/sort.c
[ 38%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/timers.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/timers.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/timers.c
[ 40%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/tokenizer.c
[ 41%] Building C object libmetis/CMakeFiles/metis.dir/__/GKlib/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/__/GKlib/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/util.c
[ 43%] Building C object libmetis/CMakeFiles/metis.dir/auxapi.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/auxapi.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/auxapi.c
[ 45%] Building C object libmetis/CMakeFiles/metis.dir/balance.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/balance.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/balance.c
[ 46%] Building C object libmetis/CMakeFiles/metis.dir/bucketsort.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/bucketsort.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/bucketsort.c
[ 48%] Building C object libmetis/CMakeFiles/metis.dir/checkgraph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/checkgraph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/checkgraph.c
[ 50%] Building C object libmetis/CMakeFiles/metis.dir/coarsen.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/coarsen.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/coarsen.c
[ 51%] Building C object libmetis/CMakeFiles/metis.dir/compress.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/compress.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/compress.c
[ 53%] Building C object libmetis/CMakeFiles/metis.dir/contig.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/contig.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c
[ 54%] Building C object libmetis/CMakeFiles/metis.dir/debug.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/debug.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c
[ 56%] Building C object libmetis/CMakeFiles/metis.dir/fm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/fm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/fm.c
[ 58%] Building C object libmetis/CMakeFiles/metis.dir/fortran.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/fortran.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/fortran.c
[ 59%] Building C object libmetis/CMakeFiles/metis.dir/frename.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/frename.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/frename.c
[ 61%] Building C object libmetis/CMakeFiles/metis.dir/gklib.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/gklib.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c
[ 62%] Building C object libmetis/CMakeFiles/metis.dir/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/graph.c
[ 64%] Building C object libmetis/CMakeFiles/metis.dir/initpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/initpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c
[ 66%] Building C object libmetis/CMakeFiles/metis.dir/kmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kmetis.c
[ 67%] Building C object libmetis/CMakeFiles/metis.dir/kwayfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kwayfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c
[ 69%] Building C object libmetis/CMakeFiles/metis.dir/kwayrefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/kwayrefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayrefine.c
[ 70%] Building C object libmetis/CMakeFiles/metis.dir/mcutil.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mcutil.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mcutil.c
[ 72%] Building C object libmetis/CMakeFiles/metis.dir/mesh.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mesh.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mesh.c
[ 74%] Building C object libmetis/CMakeFiles/metis.dir/meshpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/meshpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/meshpart.c
[ 75%] Building C object libmetis/CMakeFiles/metis.dir/minconn.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/minconn.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c
[ 77%] Building C object libmetis/CMakeFiles/metis.dir/mincover.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mincover.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mincover.c
[ 79%] Building C object libmetis/CMakeFiles/metis.dir/mmd.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/mmd.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/mmd.c
[ 80%] Building C object libmetis/CMakeFiles/metis.dir/ometis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/ometis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c
[ 82%] Building C object libmetis/CMakeFiles/metis.dir/options.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/options.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/options.c
[ 83%] Building C object libmetis/CMakeFiles/metis.dir/parmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/parmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/parmetis.c
[ 85%] Building C object libmetis/CMakeFiles/metis.dir/pmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/pmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/pmetis.c
[ 87%] Building C object libmetis/CMakeFiles/metis.dir/refine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/refine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/refine.c
[ 88%] Building C object libmetis/CMakeFiles/metis.dir/separator.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/separator.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/separator.c
[ 90%] Building C object libmetis/CMakeFiles/metis.dir/sfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/sfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/sfm.c
[ 91%] Building C object libmetis/CMakeFiles/metis.dir/srefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/srefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/srefine.c
[ 93%] Building C object libmetis/CMakeFiles/metis.dir/stat.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/stat.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/stat.c
[ 95%] Building C object libmetis/CMakeFiles/metis.dir/timing.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/timing.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/timing.c
[ 96%] Building C object libmetis/CMakeFiles/metis.dir/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/util.c
[ 98%] Building C object libmetis/CMakeFiles/metis.dir/wspace.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/mpicc -Dmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/. -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/include  -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE  -fPIC   -o CMakeFiles/metis.dir/wspace.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/wspace.c
[100%] Linking C shared library libmetis.so
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis && /usr/bin/cmake -E cmake_link_script CMakeFiles/metis.dir/link.txt --verbose=1
/usr/bin/mpicc -fPIC -fPIC -fopenmp -g -O -DLINUX -D_FILE_OFFSET_BITS=64 -std=c99 -fno-strict-aliasing -fPIC -Wall -pedantic -Wno-unused-variable -Wno-unknown-pragmas -DNDEBUG -DNDEBUG2 -DUSE_GKRAND -DHAVE_EXECINFO_H -DHAVE_GETLINE   -shared -Wl,-soname,libmetis.so -o libmetis.so CMakeFiles/metis.dir/__/GKlib/b64.c.o CMakeFiles/metis.dir/__/GKlib/blas.c.o CMakeFiles/metis.dir/__/GKlib/csr.c.o CMakeFiles/metis.dir/__/GKlib/error.c.o CMakeFiles/metis.dir/__/GKlib/evaluate.c.o CMakeFiles/metis.dir/__/GKlib/fkvkselect.c.o CMakeFiles/metis.dir/__/GKlib/fs.c.o CMakeFiles/metis.dir/__/GKlib/getopt.c.o CMakeFiles/metis.dir/__/GKlib/gkregex.c.o CMakeFiles/metis.dir/__/GKlib/graph.c.o CMakeFiles/metis.dir/__/GKlib/htable.c.o CMakeFiles/metis.dir/__/GKlib/io.c.o CMakeFiles/metis.dir/__/GKlib/itemsets.c.o CMakeFiles/metis.dir/__/GKlib/mcore.c.o CMakeFiles/metis.dir/__/GKlib/memory.c.o CMakeFiles/metis.dir/__/GKlib/omp.c.o CMakeFiles/metis.dir/__/GKlib/pdb.c.o CMakeFiles/metis.dir/__/GKlib/pqueue.c.o CMakeFiles/metis.dir/__/GKlib/random.c.o CMakeFiles/metis.dir/__/GKlib/rw.c.o CMakeFiles/metis.dir/__/GKlib/seq.c.o CMakeFiles/metis.dir/__/GKlib/sort.c.o CMakeFiles/metis.dir/__/GKlib/string.c.o CMakeFiles/metis.dir/__/GKlib/timers.c.o CMakeFiles/metis.dir/__/GKlib/tokenizer.c.o CMakeFiles/metis.dir/__/GKlib/util.c.o CMakeFiles/metis.dir/auxapi.c.o CMakeFiles/metis.dir/balance.c.o CMakeFiles/metis.dir/bucketsort.c.o CMakeFiles/metis.dir/checkgraph.c.o CMakeFiles/metis.dir/coarsen.c.o CMakeFiles/metis.dir/compress.c.o CMakeFiles/metis.dir/contig.c.o CMakeFiles/metis.dir/debug.c.o CMakeFiles/metis.dir/fm.c.o CMakeFiles/metis.dir/fortran.c.o CMakeFiles/metis.dir/frename.c.o CMakeFiles/metis.dir/gklib.c.o CMakeFiles/metis.dir/graph.c.o CMakeFiles/metis.dir/initpart.c.o CMakeFiles/metis.dir/kmetis.c.o CMakeFiles/metis.dir/kwayfm.c.o CMakeFiles/metis.dir/kwayrefine.c.o CMakeFiles/metis.dir/mcutil.c.o CMakeFiles/metis.dir/mesh.c.o CMakeFiles/metis.dir/meshpart.c.o CMakeFiles/metis.dir/minconn.c.o CMakeFiles/metis.dir/mincover.c.o CMakeFiles/metis.dir/mmd.c.o CMakeFiles/metis.dir/ometis.c.o CMakeFiles/metis.dir/options.c.o CMakeFiles/metis.dir/parmetis.c.o CMakeFiles/metis.dir/pmetis.c.o CMakeFiles/metis.dir/refine.c.o CMakeFiles/metis.dir/separator.c.o CMakeFiles/metis.dir/sfm.c.o CMakeFiles/metis.dir/srefine.c.o CMakeFiles/metis.dir/stat.c.o CMakeFiles/metis.dir/timing.c.o CMakeFiles/metis.dir/util.c.o CMakeFiles/metis.dir/wspace.c.o -lm 
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[100%] Built target metis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles 0
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/libmetis/CMakeFiles/metis.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/make -f libmetis/CMakeFiles/metis.dir/build.make libmetis/CMakeFiles/metis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
make[2]: Nothing to be done for 'libmetis/CMakeFiles/metis.dir/build'.
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
[100%] Built target metis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build/CMakeFiles 0
/usr/bin/make -f CMakeFiles/Makefile2 preinstall
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
make[1]: Nothing to be done for 'preinstall'.
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/petsc-build'
Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_tls.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.so
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_defs.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/gklib_rename.h/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c: In function ???gk_csr_Normalize???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c:1344:9: warning: this ???if??? clause does not guard... [-Wmisleading-indentation]
    else if (norm == 1)
         ^~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c:1346:11: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ???if???
           for (j=ptr[i]; j<ptr[i+1]; j++)
           ^~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c:1371:9: warning: this ???if??? clause does not guard... [-Wmisleading-indentation]
    else if (norm == 1)
         ^~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/csr.c:1373:11: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ???if???
           for (j=ptr[i]; j<ptr[i+1]; j++)
           ^~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/error.c: In function ???gk_strerror???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/error.c:183:3: warning: implicit declaration of function ???strerror_r???; did you mean ???strerror???? [-Wimplicit-function-declaration]
   strerror_r(errnum, buf, 1024);
   ^~~~~~~~~~
   strerror
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c: In function ???gk_getopt_internal???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c:343:5: warning: this ???if??? clause does not guard... [-Wmisleading-indentation]
     if (gk_optind == 0)
     ^~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c:345:7: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ???if???
       optstring = gk_getopt_initialize (argc, argv, optstring);
       ^~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c:700:2: warning: this ???else??? clause does not guard... [-Wmisleading-indentation]
  else
  ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/getopt.c:703:4: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ???else???
    nextchar = NULL;
    ^~~~~~~~
In file included from /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/GKlib.h:74,
                 from /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:12:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_ipqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:19:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_ipq,   gk_ipq_t,   gk_ikv_t,   int,      gk_idx_t, gk_ikvmalloc,   INT_MAX,    key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_i32pqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:20:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_i32pq, gk_i32pq_t, gk_i32kv_t, int32_t,  gk_idx_t, gk_i32kvmalloc, INT32_MAX,  key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_i64pqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:21:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_i64pq, gk_i64pq_t, gk_i64kv_t, int64_t,  gk_idx_t, gk_i64kvmalloc, INT64_MAX,  key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_fpqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:22:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_fpq,   gk_fpq_t,   gk_fkv_t,   float,    gk_idx_t, gk_fkvmalloc,   FLT_MAX,    key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_dpqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:23:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_dpq,   gk_dpq_t,   gk_dkv_t,   double,   gk_idx_t, gk_dkvmalloc,   DBL_MAX,    key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c: In function ???gk_idxpqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/pqueue.c:24:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(gk_idxpq, gk_idxpq_t, gk_idxkv_t, gk_idx_t, gk_idx_t, gk_idxkvmalloc, GK_IDX_MAX, key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/coarsen.c: In function ???libmetis__Match_RM???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/coarsen.c:152:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *maxvwgt;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c: In function ???libmetis__EliminateComponents???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c:345:11: warning: variable ???tpwgts??? set but not used [-Wunused-but-set-variable]
   real_t *tpwgts;
           ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c: In function ???libmetis__MoveGroupContigForCut???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c:534:34: warning: variable ???nvtxs??? set but not used [-Wunused-but-set-variable]
   idx_t i, ii, iii, j, jj, k, l, nvtxs, nbnd, from, me;
                                  ^~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c: In function ???libmetis__MoveGroupContigForVol???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/contig.c:605:34: warning: variable ???nvtxs??? set but not used [-Wunused-but-set-variable]
   idx_t i, ii, iii, j, jj, k, l, nvtxs, from, me, other, xgain;
                                  ^~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__CheckBnd???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:124:43: warning: variable ???bndind??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                           ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:124:34: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                  ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__CheckBnd2???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:161:43: warning: variable ???bndind??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                           ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:161:34: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                  ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__CheckNodeBnd???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:198:43: warning: variable ???bndind??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                           ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:198:34: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                                  ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:198:17: warning: variable ???adjncy??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:198:10: warning: variable ???xadj??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where, *bndptr, *bndind;
          ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__CheckRInfo???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:235:11: warning: variable ???nbrs??? set but not used [-Wunused-but-set-variable]
   cnbr_t *nbrs;
           ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__CheckNodePartitionParams???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:258:26: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *adjwgt, *vwgt, *where;
                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c: In function ???libmetis__IsSeparable???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:312:17: warning: variable ???adjncy??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *where;
                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/debug.c:311:22: warning: variable ???other??? set but not used [-Wunused-but-set-variable]
   idx_t i, j, nvtxs, other;
                      ^~~~~
In file included from /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/GKlib.h:74,
                 from /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/metislib.h:17,
                 from /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c:12:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c: In function ???libmetis__ipqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c:33:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(ipq, ipq_t, ikv_t, idx_t, idx_t, ikvmalloc, IDX_MAX, key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c: In function ???libmetis__rpqCheckHeap???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/GKlib/gk_mkpqueue.h:386:8: warning: variable ???heap??? set but not used [-Wunused-but-set-variable]
   KVT *heap;\
        ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/gklib.c:34:1: note: in expansion of macro ???GK_MKPQUEUE???
 GK_MKPQUEUE(rpq, rpq_t, rkv_t, real_t, idx_t, rkvmalloc, REAL_MAX, key_gt)
 ^~~~~~~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c: In function ???libmetis__RandomBisection???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c:119:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *where;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c:119:24: warning: variable ???adjncy??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *where;
                        ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c:119:10: warning: variable ???xadj??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *where;
          ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c: In function ???libmetis__GrowBisection???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c:195:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *where;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c: In function ???libmetis__GrowBisectionNode???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/initpart.c:438:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *where, *bndind;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c: In function ???libmetis__Greedy_KWayCutOptimize???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:69:74: warning: variable ???adwgts??? set but not used [-Wunused-but-set-variable]
   idx_t maxndoms, *safetos=NULL, *nads=NULL, *doms=NULL, **adids=NULL, **adwgts=NULL;
                                                                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c: In function ???libmetis__Greedy_KWayVolOptimize???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:379:74: warning: variable ???adwgts??? set but not used [-Wunused-but-set-variable]
   idx_t maxndoms, *safetos=NULL, *nads=NULL, *doms=NULL, **adids=NULL, **adwgts=NULL;
                                                                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:377:33: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *where, *pwgts, *perm, *bndptr, *bndind, *minwgt, *maxwgt, *itpwgts;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c: In function ???libmetis__Greedy_McKWayCutOptimize???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:693:74: warning: variable ???adwgts??? set but not used [-Wunused-but-set-variable]
   idx_t maxndoms, *safetos=NULL, *nads=NULL, *doms=NULL, **adids=NULL, **adwgts=NULL;
                                                                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c: In function ???libmetis__Greedy_McKWayVolOptimize???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:1035:74: warning: variable ???adwgts??? set but not used [-Wunused-but-set-variable]
   idx_t maxndoms, *safetos=NULL, *nads=NULL, *doms=NULL, **adids=NULL, **adwgts=NULL;
                                                                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayfm.c:1033:33: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *where, *pwgts, *perm, *bndptr, *bndind, *minwgt, *maxwgt;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayrefine.c: In function ???libmetis__ComputeKWayVolGains???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/kwayrefine.c:565:34: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vsize, *adjncy, *adjwgt, *where,
                                  ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c: In function ???libmetis__ComputeSubDomainGraph???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c:21:26: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *adjwgt, *where;
                          ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c:21:17: warning: variable ???adjncy??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *adjwgt, *where;
                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c:21:10: warning: variable ???xadj??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *adjncy, *adjwgt, *where;
          ^~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c: In function ???libmetis__MoveGroupMinConnForCut???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c:480:29: warning: variable ???nvtxs??? set but not used [-Wunused-but-set-variable]
   idx_t i, ii, j, jj, k, l, nvtxs, nbnd, from, me;
                             ^~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c: In function ???libmetis__MoveGroupMinConnForVol???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/minconn.c:564:29: warning: variable ???nvtxs??? set but not used [-Wunused-but-set-variable]
   idx_t i, ii, j, jj, k, l, nvtxs, from, me, other, xgain, ewgt;
                             ^~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c: In function ???libmetis__MlevelNestedDissection???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c:186:15: warning: variable ???nvtxs??? set but not used [-Wunused-but-set-variable]
   idx_t i, j, nvtxs, nbnd;
               ^~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c: In function ???libmetis__SplitGraphOrder???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c:426:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *label, *where, *bndptr, *bndind;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c: In function ???libmetis__SplitGraphOrderCC???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/ometis.c:556:33: warning: variable ???adjwgt??? set but not used [-Wunused-but-set-variable]
   idx_t *xadj, *vwgt, *adjncy, *adjwgt, *label, *where, *bndptr, *bndind;
                                 ^~~~~~
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/separator.c: In function ???libmetis__ConstructMinCoverSeparator???:
/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.metis/libmetis/separator.c:73:27: warning: variable ???bndptr??? set but not used [-Wunused-but-set-variable]
   idx_t *where, *bndind, *bndptr, *vmap, *ivmap, *cover;
                           ^~~~~~
********End of Output of running make on METIS *******
              Checking for library in Download METIS: ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a']
              Contents: ['share', 'bin', 'externalpackages', 'lib', 'include']
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                  Checking for functions [METIS_PartGraphKway] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a'] ['libm.a']
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char METIS_PartGraphKway();
static void _check_METIS_PartGraphKway() { METIS_PartGraphKway(); }

int main() {
_check_METIS_PartGraphKway();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lmetis -lm -lstdc++ -ldl 
                    Defined "HAVE_LIBMETIS" to "1"
                    Popping language C
                  No functions to check for in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a'] ['libm.a']
              Checking for headers Download METIS: ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                  Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['metis.h'] in ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                    Checking include with compiler flags var CPPFLAGS ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.headers  -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/usr/include -I/usr/lib/openmpi /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h" 1
# 94 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
# 1 "/usr/include/inttypes.h" 1 3 4
# 25 "/usr/include/inttypes.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/inttypes.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 1 3 4
# 9 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 26 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 27 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
# 30 "/usr/include/bits/types.h" 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wchar.h" 1 3 4
# 29 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 30 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 35 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 24 "/usr/include/bits/stdint-uintn.h" 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 38 "/usr/include/stdint.h" 2 3 4
typedef __int_least8_t int_least8_t;
typedef __int_least16_t int_least16_t;
typedef __int_least32_t int_least32_t;
typedef __int_least64_t int_least64_t;
typedef __uint_least8_t uint_least8_t;
typedef __uint_least16_t uint_least16_t;
typedef __uint_least32_t uint_least32_t;
typedef __uint_least64_t uint_least64_t;
typedef signed char int_fast8_t;
typedef long int int_fast16_t;
typedef long int int_fast32_t;
typedef long int int_fast64_t;
# 71 "/usr/include/stdint.h" 3 4
typedef unsigned char uint_fast8_t;
typedef unsigned long int uint_fast16_t;
typedef unsigned long int uint_fast32_t;
typedef unsigned long int uint_fast64_t;
# 87 "/usr/include/stdint.h" 3 4
typedef long int intptr_t;
typedef unsigned long int uintptr_t;
# 101 "/usr/include/stdint.h" 3 4
typedef __intmax_t intmax_t;
typedef __uintmax_t uintmax_t;
# 10 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 2 3 4
# 28 "/usr/include/inttypes.h" 2 3 4
typedef int __gwchar_t;
# 266 "/usr/include/inttypes.h" 3 4
typedef struct
  {
    long int quot;
    long int rem;
  } imaxdiv_t;
# 290 "/usr/include/inttypes.h" 3 4
extern intmax_t imaxabs (intmax_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern imaxdiv_t imaxdiv (intmax_t __numer, intmax_t __denom)
      __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern intmax_t strtoimax (const char *__restrict __nptr,
      char **__restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t strtoumax (const char *__restrict __nptr,
       char ** __restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern intmax_t wcstoimax (const __gwchar_t *__restrict __nptr,
      __gwchar_t **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t wcstoumax (const __gwchar_t *__restrict __nptr,
       __gwchar_t ** __restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
# 432 "/usr/include/inttypes.h" 3 4
# 95 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h" 2
# 103 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
  
# 103 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
 typedef int32_t idx_t;
# 134 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
  typedef float real_t;
# 199 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
int METIS_PartGraphRecursive(idx_t *nvtxs, idx_t *ncon, idx_t *xadj,
                  idx_t *adjncy, idx_t *vwgt, idx_t *vsize, idx_t *adjwgt,
                  idx_t *nparts, real_t *tpwgts, real_t *ubvec, idx_t *options,
                  idx_t *edgecut, idx_t *part);
int METIS_PartGraphKway(idx_t *nvtxs, idx_t *ncon, idx_t *xadj,
                  idx_t *adjncy, idx_t *vwgt, idx_t *vsize, idx_t *adjwgt,
                  idx_t *nparts, real_t *tpwgts, real_t *ubvec, idx_t *options,
                  idx_t *edgecut, idx_t *part);
int METIS_MeshToDual(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *ncommon, idx_t *numflag, idx_t **r_xadj, idx_t **r_adjncy);
int METIS_MeshToNodal(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *numflag, idx_t **r_xadj, idx_t **r_adjncy);
int METIS_PartMeshNodal(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *vwgt, idx_t *vsize, idx_t *nparts, real_t *tpwgts,
                  idx_t *options, idx_t *objval, idx_t *epart, idx_t *npart);
int METIS_PartMeshDual(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *vwgt, idx_t *vsize, idx_t *ncommon, idx_t *nparts,
                  real_t *tpwgts, idx_t *options, idx_t *objval, idx_t *epart,
                  idx_t *npart);
int METIS_NodeND(idx_t *nvtxs, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
                  idx_t *options, idx_t *perm, idx_t *iperm);
int METIS_Free(void *ptr);
int METIS_SetDefaultOptions(idx_t *options);
int METIS_NodeNDP(idx_t nvtxs, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
                   idx_t npes, idx_t *options, idx_t *perm, idx_t *iperm,
                   idx_t *sizes);
int METIS_ComputeVertexSeparator(idx_t *nvtxs, idx_t *xadj, idx_t *adjncy,
                   idx_t *vwgt, idx_t *options, idx_t *sepsize, idx_t *part);
int METIS_NodeRefine(idx_t nvtxs, idx_t *xadj, idx_t *vwgt, idx_t *adjncy,
                   idx_t *where, idx_t *hmarker, real_t ubfactor);
# 255 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
typedef enum {
  METIS_OK = 1,
  METIS_ERROR_INPUT = -2,
  METIS_ERROR_MEMORY = -3,
  METIS_ERROR = -4
} rstatus_et;
typedef enum {
  METIS_OP_PMETIS,
  METIS_OP_KMETIS,
  METIS_OP_OMETIS
} moptype_et;
typedef enum {
  METIS_OPTION_PTYPE,
  METIS_OPTION_OBJTYPE,
  METIS_OPTION_CTYPE,
  METIS_OPTION_IPTYPE,
  METIS_OPTION_RTYPE,
  METIS_OPTION_DBGLVL,
  METIS_OPTION_NITER,
  METIS_OPTION_NCUTS,
  METIS_OPTION_SEED,
  METIS_OPTION_NO2HOP,
  METIS_OPTION_MINCONN,
  METIS_OPTION_CONTIG,
  METIS_OPTION_COMPRESS,
  METIS_OPTION_CCORDER,
  METIS_OPTION_PFACTOR,
  METIS_OPTION_NSEPS,
  METIS_OPTION_UFACTOR,
  METIS_OPTION_NUMBERING,
  METIS_OPTION_HELP,
  METIS_OPTION_TPWGTS,
  METIS_OPTION_NCOMMON,
  METIS_OPTION_NOOUTPUT,
  METIS_OPTION_BALANCE,
  METIS_OPTION_GTYPE,
  METIS_OPTION_UBVEC
} moptions_et;
typedef enum {
  METIS_PTYPE_RB,
  METIS_PTYPE_KWAY
} mptype_et;
typedef enum {
  METIS_GTYPE_DUAL,
  METIS_GTYPE_NODAL
} mgtype_et;
typedef enum {
  METIS_CTYPE_RM,
  METIS_CTYPE_SHEM
} mctype_et;
typedef enum {
  METIS_IPTYPE_GROW,
  METIS_IPTYPE_RANDOM,
  METIS_IPTYPE_EDGE,
  METIS_IPTYPE_NODE,
  METIS_IPTYPE_METISRB
} miptype_et;
typedef enum {
  METIS_RTYPE_FM,
  METIS_RTYPE_GREEDY,
  METIS_RTYPE_SEP2SIDED,
  METIS_RTYPE_SEP1SIDED
} mrtype_et;
typedef enum {
  METIS_DBG_INFO = 1,
  METIS_DBG_TIME = 2,
  METIS_DBG_COARSEN = 4,
  METIS_DBG_REFINE = 8,
  METIS_DBG_IPART = 16,
  METIS_DBG_MOVEINFO = 32,
  METIS_DBG_SEPINFO = 64,
  METIS_DBG_CONNINFO = 128,
  METIS_DBG_CONTIGINFO = 256,
  METIS_DBG_MEMORY = 2048,
} mdbglvl_et;
typedef enum {
  METIS_OBJTYPE_CUT,
  METIS_OBJTYPE_VOL,
  METIS_OBJTYPE_NODE
} mobjtype_et;
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['metis.h'] in ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                  Popping language C
                        All intermediate test results are stored in /tmp/petsc-VuReVk/config.packages.metis
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.packages.metis/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.packages.metis -fPIC -fopenmp   -g -O  -I/home/ryan/projects/moose/petsc/linux-c-opt/include /tmp/petsc-VuReVk/config.packages.metis/conftest.c 
Possible ERROR while running compiler:
stderr:
/tmp/petsc-VuReVk/config.packages.metis/conftest.c: In function ???main???:
/tmp/petsc-VuReVk/config.packages.metis/conftest.c:8:7: warning: extra tokens at end of #endif directive [-Wendif-labels]
 #endif;
       ^
Source:
#include "confdefs.h"
#include "conffix.h"
#include "metis.h"

int main() {
#if (IDXTYPEWIDTH != 32)
#error incompatible IDXTYPEWIDTH
#endif;
  return 0;
}
================================================================================
TEST checkSharedLibrary from config.packages.metis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.metis(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.pragmatic(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.pragmatic(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.parmetis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.parmetis(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.parmetis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/parmetis.py:39)
TESTING: configureLibrary from config.packages.parmetis(config/BuildSystem/config/packages/parmetis.py:39)
==================================================================================
              Checking for a functional parmetis
                      Looking for PARMETIS at git.parmetis, hg.parmetis or a directory starting with ['petsc-pkg-parmetis']
                      Found a copy of PARMETIS in git.parmetis
Executing: ['git', 'rev-parse', '--git-dir']
stdout: .git
Executing: ['git', 'cat-file', '-e', 'v4.0.3-p4^{commit}']
Executing: ['git', 'rev-parse', 'v4.0.3-p4']
stdout: 73dab469aa363afcdce33cde3e5575554d7ce78c
Executing: ['git', 'stash']
stdout: No local changes to save
Executing: ['git', 'clean', '-f', '-d', '-x']
stdout:
Removing parmetis.petscconf
Removing pkg.gitcommit
Executing: ['git', 'checkout', '-f', '73dab469aa363afcdce33cde3e5575554d7ce78c']
                            Pushing language C
                            Popping language C
                            Pushing language Cxx
                            Popping language Cxx
                            Pushing language FC
                            Popping language FC
Have to rebuild PARMETIS, /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/parmetis.petscconf != /home/ryan/projects/moose/petsc/linux-c-opt/lib/petsc/conf/pkg.conf.parmetis
                          ===============================================================================
                              Configuring PARMETIS with cmake, this may take several minutes
                          ===============================================================================
                        
Executing: cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/home/ryan/projects/moose/petsc/linux-c-opt -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_COMPILER="mpicxx" -DCMAKE_CXX_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_COMPILER="mpif90" -DCMAKE_Fortran_FLAGS:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -fopenmp -g -O" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -fopenmp -g -O" -DBUILD_SHARED_LIBS=on -DGKLIB_PATH=../headers -DMETIS_PATH=/home/ryan/projects/moose/petsc/linux-c-opt -DSHARED=1 -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON
stdout:
-- The C compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build
                          ===============================================================================
                              Compiling and installing PARMETIS; this may take several minutes
                          ===============================================================================
                        
Executing: cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/make -j4 -l6.0  &&  /usr/bin/make install
stdout:
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis/CMakeFiles/parmetis.dir/DependInfo.cmake --color=
Scanning dependencies of target parmetis
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[  2%] Building C object libparmetis/CMakeFiles/parmetis.dir/ametis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ametis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ametis.c
[  4%] Building C object libparmetis/CMakeFiles/parmetis.dir/akwayfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/akwayfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/akwayfm.c
[  7%] Building C object libparmetis/CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ComputeVertexSeparator.c
[  9%] Building C object libparmetis/CMakeFiles/parmetis.dir/balancemylink.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/balancemylink.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/balancemylink.c
[ 12%] Building C object libparmetis/CMakeFiles/parmetis.dir/comm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/comm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/comm.c
[ 14%] Building C object libparmetis/CMakeFiles/parmetis.dir/csrmatch.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/csrmatch.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/csrmatch.c
[ 17%] Building C object libparmetis/CMakeFiles/parmetis.dir/ctrl.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ctrl.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ctrl.c
[ 19%] Building C object libparmetis/CMakeFiles/parmetis.dir/debug.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/debug.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/debug.c
[ 21%] Building C object libparmetis/CMakeFiles/parmetis.dir/diffutil.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/diffutil.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/diffutil.c
[ 24%] Building C object libparmetis/CMakeFiles/parmetis.dir/frename.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/frename.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/frename.c
[ 26%] Building C object libparmetis/CMakeFiles/parmetis.dir/gkmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/gkmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/gkmetis.c
[ 29%] Building C object libparmetis/CMakeFiles/parmetis.dir/gkmpi.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/gkmpi.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/gkmpi.c
[ 31%] Building C object libparmetis/CMakeFiles/parmetis.dir/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/graph.c
[ 34%] Building C object libparmetis/CMakeFiles/parmetis.dir/initbalance.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initbalance.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initbalance.c
[ 36%] Building C object libparmetis/CMakeFiles/parmetis.dir/initmsection.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initmsection.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initmsection.c
[ 39%] Building C object libparmetis/CMakeFiles/parmetis.dir/initpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initpart.c
[ 41%] Building C object libparmetis/CMakeFiles/parmetis.dir/kmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/kmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/kmetis.c
[ 43%] Building C object libparmetis/CMakeFiles/parmetis.dir/kwayrefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/kwayrefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/kwayrefine.c
[ 46%] Building C object libparmetis/CMakeFiles/parmetis.dir/match.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/match.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/match.c
[ 48%] Building C object libparmetis/CMakeFiles/parmetis.dir/mdiffusion.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mdiffusion.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mdiffusion.c
[ 51%] Building C object libparmetis/CMakeFiles/parmetis.dir/mesh.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mesh.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mesh.c
[ 53%] Building C object libparmetis/CMakeFiles/parmetis.dir/mmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mmetis.c
[ 56%] Building C object libparmetis/CMakeFiles/parmetis.dir/move.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/move.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/move.c
[ 58%] Building C object libparmetis/CMakeFiles/parmetis.dir/msetup.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/msetup.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/msetup.c
[ 60%] Building C object libparmetis/CMakeFiles/parmetis.dir/node_refine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/node_refine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/node_refine.c
[ 63%] Building C object libparmetis/CMakeFiles/parmetis.dir/ometis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ometis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ometis.c
[ 65%] Building C object libparmetis/CMakeFiles/parmetis.dir/pspases.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/pspases.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/pspases.c
[ 68%] Building C object libparmetis/CMakeFiles/parmetis.dir/redomylink.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/redomylink.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/redomylink.c
[ 70%] Building C object libparmetis/CMakeFiles/parmetis.dir/remap.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/remap.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/remap.c
[ 73%] Building C object libparmetis/CMakeFiles/parmetis.dir/renumber.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/renumber.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/renumber.c
[ 75%] Building C object libparmetis/CMakeFiles/parmetis.dir/rmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/rmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/rmetis.c
[ 78%] Building C object libparmetis/CMakeFiles/parmetis.dir/serial.c.o
[ 80%] Building C object libparmetis/CMakeFiles/parmetis.dir/selectq.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/serial.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/serial.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/selectq.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/selectq.c
[ 82%] Building C object libparmetis/CMakeFiles/parmetis.dir/timer.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/timer.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/timer.c
[ 85%] Building C object libparmetis/CMakeFiles/parmetis.dir/stat.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/stat.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/stat.c
[ 87%] Building C object libparmetis/CMakeFiles/parmetis.dir/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/util.c
[ 90%] Building C object libparmetis/CMakeFiles/parmetis.dir/wave.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/wave.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/wave.c
[ 92%] Building C object libparmetis/CMakeFiles/parmetis.dir/weird.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/weird.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/weird.c
[ 95%] Building C object libparmetis/CMakeFiles/parmetis.dir/wspace.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/wspace.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/wspace.c
[ 97%] Building C object libparmetis/CMakeFiles/parmetis.dir/xyzpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/xyzpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/xyzpart.c
[100%] Linking C shared library libparmetis.so
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/cmake -E cmake_link_script CMakeFiles/parmetis.dir/link.txt --verbose=1
/usr/bin/mpicc -fPIC -fPIC -fopenmp -g -O  -shared -Wl,-soname,libparmetis.so -o libparmetis.so CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o CMakeFiles/parmetis.dir/akwayfm.c.o CMakeFiles/parmetis.dir/ametis.c.o CMakeFiles/parmetis.dir/balancemylink.c.o CMakeFiles/parmetis.dir/comm.c.o CMakeFiles/parmetis.dir/csrmatch.c.o CMakeFiles/parmetis.dir/ctrl.c.o CMakeFiles/parmetis.dir/debug.c.o CMakeFiles/parmetis.dir/diffutil.c.o CMakeFiles/parmetis.dir/frename.c.o CMakeFiles/parmetis.dir/gkmetis.c.o CMakeFiles/parmetis.dir/gkmpi.c.o CMakeFiles/parmetis.dir/graph.c.o CMakeFiles/parmetis.dir/initbalance.c.o CMakeFiles/parmetis.dir/initmsection.c.o CMakeFiles/parmetis.dir/initpart.c.o CMakeFiles/parmetis.dir/kmetis.c.o CMakeFiles/parmetis.dir/kwayrefine.c.o CMakeFiles/parmetis.dir/match.c.o CMakeFiles/parmetis.dir/mdiffusion.c.o CMakeFiles/parmetis.dir/mesh.c.o CMakeFiles/parmetis.dir/mmetis.c.o CMakeFiles/parmetis.dir/move.c.o CMakeFiles/parmetis.dir/msetup.c.o CMakeFiles/parmetis.dir/node_refine.c.o CMakeFiles/parmetis.dir/ometis.c.o CMakeFiles/parmetis.dir/pspases.c.o CMakeFiles/parmetis.dir/redomylink.c.o CMakeFiles/parmetis.dir/remap.c.o CMakeFiles/parmetis.dir/renumber.c.o CMakeFiles/parmetis.dir/rmetis.c.o CMakeFiles/parmetis.dir/selectq.c.o CMakeFiles/parmetis.dir/serial.c.o CMakeFiles/parmetis.dir/stat.c.o CMakeFiles/parmetis.dir/timer.c.o CMakeFiles/parmetis.dir/util.c.o CMakeFiles/parmetis.dir/wave.c.o CMakeFiles/parmetis.dir/weird.c.o CMakeFiles/parmetis.dir/wspace.c.o CMakeFiles/parmetis.dir/xyzpart.c.o  -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib: -lmetis 
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[100%] Built target parmetis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles 0
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis/CMakeFiles/parmetis.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
make[2]: Nothing to be done for 'libparmetis/CMakeFiles/parmetis.dir/build'.
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[100%] Built target parmetis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles 0
/usr/bin/make -f CMakeFiles/Makefile2 preinstall
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
make[1]: Nothing to be done for 'preinstall'.
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.so
-- Set runtime path of "/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.so" to "/home/ryan/projects/moose/petsc/linux-c-opt/lib"
********Output of running make on PARMETIS follows *******
-- The C compiler identification is GNU 8.2.1
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-buildCMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS
    CMAKE_CXX_FLAGS_DEBUG
    CMAKE_CXX_FLAGS_RELEASE
    CMAKE_C_FLAGS_DEBUG
    CMAKE_C_FLAGS_RELEASE
    CMAKE_Fortran_COMPILER
    CMAKE_Fortran_FLAGS
    CMAKE_Fortran_FLAGS_DEBUG
    CMAKE_Fortran_FLAGS_RELEASE


/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis/CMakeFiles/parmetis.dir/DependInfo.cmake --color=
Scanning dependencies of target parmetis
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[  2%] Building C object libparmetis/CMakeFiles/parmetis.dir/ametis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ametis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ametis.c
[  4%] Building C object libparmetis/CMakeFiles/parmetis.dir/akwayfm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/akwayfm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/akwayfm.c
[  7%] Building C object libparmetis/CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ComputeVertexSeparator.c
[  9%] Building C object libparmetis/CMakeFiles/parmetis.dir/balancemylink.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/balancemylink.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/balancemylink.c
[ 12%] Building C object libparmetis/CMakeFiles/parmetis.dir/comm.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/comm.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/comm.c
[ 14%] Building C object libparmetis/CMakeFiles/parmetis.dir/csrmatch.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/csrmatch.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/csrmatch.c
[ 17%] Building C object libparmetis/CMakeFiles/parmetis.dir/ctrl.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ctrl.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ctrl.c
[ 19%] Building C object libparmetis/CMakeFiles/parmetis.dir/debug.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/debug.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/debug.c
[ 21%] Building C object libparmetis/CMakeFiles/parmetis.dir/diffutil.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/diffutil.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/diffutil.c
[ 24%] Building C object libparmetis/CMakeFiles/parmetis.dir/frename.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/frename.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/frename.c
[ 26%] Building C object libparmetis/CMakeFiles/parmetis.dir/gkmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/gkmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/gkmetis.c
[ 29%] Building C object libparmetis/CMakeFiles/parmetis.dir/gkmpi.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/gkmpi.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/gkmpi.c
[ 31%] Building C object libparmetis/CMakeFiles/parmetis.dir/graph.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/graph.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/graph.c
[ 34%] Building C object libparmetis/CMakeFiles/parmetis.dir/initbalance.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initbalance.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initbalance.c
[ 36%] Building C object libparmetis/CMakeFiles/parmetis.dir/initmsection.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initmsection.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initmsection.c
[ 39%] Building C object libparmetis/CMakeFiles/parmetis.dir/initpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/initpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/initpart.c
[ 41%] Building C object libparmetis/CMakeFiles/parmetis.dir/kmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/kmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/kmetis.c
[ 43%] Building C object libparmetis/CMakeFiles/parmetis.dir/kwayrefine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/kwayrefine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/kwayrefine.c
[ 46%] Building C object libparmetis/CMakeFiles/parmetis.dir/match.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/match.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/match.c
[ 48%] Building C object libparmetis/CMakeFiles/parmetis.dir/mdiffusion.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mdiffusion.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mdiffusion.c
[ 51%] Building C object libparmetis/CMakeFiles/parmetis.dir/mesh.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mesh.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mesh.c
[ 53%] Building C object libparmetis/CMakeFiles/parmetis.dir/mmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/mmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/mmetis.c
[ 56%] Building C object libparmetis/CMakeFiles/parmetis.dir/move.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/move.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/move.c
[ 58%] Building C object libparmetis/CMakeFiles/parmetis.dir/msetup.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/msetup.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/msetup.c
[ 60%] Building C object libparmetis/CMakeFiles/parmetis.dir/node_refine.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/node_refine.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/node_refine.c
[ 63%] Building C object libparmetis/CMakeFiles/parmetis.dir/ometis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/ometis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/ometis.c
[ 65%] Building C object libparmetis/CMakeFiles/parmetis.dir/pspases.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/pspases.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/pspases.c
[ 68%] Building C object libparmetis/CMakeFiles/parmetis.dir/redomylink.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/redomylink.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/redomylink.c
[ 70%] Building C object libparmetis/CMakeFiles/parmetis.dir/remap.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/remap.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/remap.c
[ 73%] Building C object libparmetis/CMakeFiles/parmetis.dir/renumber.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/renumber.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/renumber.c
[ 75%] Building C object libparmetis/CMakeFiles/parmetis.dir/rmetis.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/rmetis.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/rmetis.c
[ 78%] Building C object libparmetis/CMakeFiles/parmetis.dir/serial.c.o
[ 80%] Building C object libparmetis/CMakeFiles/parmetis.dir/selectq.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/serial.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/serial.c
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/selectq.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/selectq.c
[ 82%] Building C object libparmetis/CMakeFiles/parmetis.dir/timer.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/timer.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/timer.c
[ 85%] Building C object libparmetis/CMakeFiles/parmetis.dir/stat.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/stat.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/stat.c
[ 87%] Building C object libparmetis/CMakeFiles/parmetis.dir/util.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/util.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/util.c
[ 90%] Building C object libparmetis/CMakeFiles/parmetis.dir/wave.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/wave.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/wave.c
[ 92%] Building C object libparmetis/CMakeFiles/parmetis.dir/weird.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/weird.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/weird.c
[ 95%] Building C object libparmetis/CMakeFiles/parmetis.dir/wspace.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/wspace.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/wspace.c
[ 97%] Building C object libparmetis/CMakeFiles/parmetis.dir/xyzpart.c.o
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/mpicc -Dparmetis_EXPORTS -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/headers -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/.  -fPIC -fopenmp -g -O -fPIC   -o CMakeFiles/parmetis.dir/xyzpart.c.o   -c /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis/xyzpart.c
[100%] Linking C shared library libparmetis.so
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis && /usr/bin/cmake -E cmake_link_script CMakeFiles/parmetis.dir/link.txt --verbose=1
/usr/bin/mpicc -fPIC -fPIC -fopenmp -g -O  -shared -Wl,-soname,libparmetis.so -o libparmetis.so CMakeFiles/parmetis.dir/ComputeVertexSeparator.c.o CMakeFiles/parmetis.dir/akwayfm.c.o CMakeFiles/parmetis.dir/ametis.c.o CMakeFiles/parmetis.dir/balancemylink.c.o CMakeFiles/parmetis.dir/comm.c.o CMakeFiles/parmetis.dir/csrmatch.c.o CMakeFiles/parmetis.dir/ctrl.c.o CMakeFiles/parmetis.dir/debug.c.o CMakeFiles/parmetis.dir/diffutil.c.o CMakeFiles/parmetis.dir/frename.c.o CMakeFiles/parmetis.dir/gkmetis.c.o CMakeFiles/parmetis.dir/gkmpi.c.o CMakeFiles/parmetis.dir/graph.c.o CMakeFiles/parmetis.dir/initbalance.c.o CMakeFiles/parmetis.dir/initmsection.c.o CMakeFiles/parmetis.dir/initpart.c.o CMakeFiles/parmetis.dir/kmetis.c.o CMakeFiles/parmetis.dir/kwayrefine.c.o CMakeFiles/parmetis.dir/match.c.o CMakeFiles/parmetis.dir/mdiffusion.c.o CMakeFiles/parmetis.dir/mesh.c.o CMakeFiles/parmetis.dir/mmetis.c.o CMakeFiles/parmetis.dir/move.c.o CMakeFiles/parmetis.dir/msetup.c.o CMakeFiles/parmetis.dir/node_refine.c.o CMakeFiles/parmetis.dir/ometis.c.o CMakeFiles/parmetis.dir/pspases.c.o CMakeFiles/parmetis.dir/redomylink.c.o CMakeFiles/parmetis.dir/remap.c.o CMakeFiles/parmetis.dir/renumber.c.o CMakeFiles/parmetis.dir/rmetis.c.o CMakeFiles/parmetis.dir/selectq.c.o CMakeFiles/parmetis.dir/serial.c.o CMakeFiles/parmetis.dir/stat.c.o CMakeFiles/parmetis.dir/timer.c.o CMakeFiles/parmetis.dir/util.c.o CMakeFiles/parmetis.dir/wave.c.o CMakeFiles/parmetis.dir/weird.c.o CMakeFiles/parmetis.dir/wspace.c.o CMakeFiles/parmetis.dir/xyzpart.c.o  -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib: -lmetis 
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[100%] Built target parmetis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles 0
/usr/bin/cmake -S/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis -B/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles/progress.marks
/usr/bin/make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/depend
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
cd /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/libparmetis/CMakeFiles/parmetis.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/make -f libparmetis/CMakeFiles/parmetis.dir/build.make libparmetis/CMakeFiles/parmetis.dir/build
make[2]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
make[2]: Nothing to be done for 'libparmetis/CMakeFiles/parmetis.dir/build'.
make[2]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
[100%] Built target parmetis
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
/usr/bin/cmake -E cmake_progress_start /home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build/CMakeFiles 0
/usr/bin/make -f CMakeFiles/Makefile2 preinstall
make[1]: Entering directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
make[1]: Nothing to be done for 'preinstall'.
make[1]: Leaving directory '/home/ryan/projects/moose/petsc/linux-c-opt/externalpackages/git.parmetis/petsc-build'
Install the project...
/usr/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h
-- Installing: /home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.so
-- Set runtime path of "/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.so" to "/home/ryan/projects/moose/petsc/linux-c-opt/lib"********End of Output of running make on PARMETIS *******
              Checking for library in Download PARMETIS: ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.a']
              Contents: ['share', 'bin', 'externalpackages', 'lib', 'include']
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                  Checking for functions [ParMETIS_V3_PartKway] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.a'] ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a', 'libm.a']
                    Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ParMETIS_V3_PartKway();
static void _check_ParMETIS_V3_PartKway() { ParMETIS_V3_PartKway(); }

int main() {
_check_ParMETIS_V3_PartKway();;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lparmetis -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lmetis -lm -lstdc++ -ldl 
                    Defined "HAVE_LIBPARMETIS" to "1"
                    Popping language C
                  No functions to check for in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.a'] ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a', 'libm.a']
              Checking for headers Download PARMETIS: ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                  Pushing language C
================================================================================
TEST checkInclude from config.headers(/home/ryan/projects/moose/petsc/config/BuildSystem/config/headers.py:86)
TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['parmetis.h'] in ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                    Checking include with compiler flags var CPPFLAGS ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi', '/home/ryan/projects/moose/petsc/linux-c-opt/include']
Executing: mpicc -E  -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.headers  -I/home/ryan/projects/moose/petsc/linux-c-opt/include -I/usr/include -I/usr/lib/openmpi -I/home/ryan/projects/moose/petsc/linux-c-opt/include /tmp/petsc-VuReVk/config.headers/conftest.c 
stdout:
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conftest.c"
# 1 "/tmp/petsc-VuReVk/config.headers/confdefs.h" 1
# 2 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/tmp/petsc-VuReVk/config.headers/conffix.h" 1
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
# 1 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h" 1
# 17 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h"
# 1 "/usr/include/mpi.h" 1 3 4
# 232 "/usr/include/mpi.h" 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
# 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long int ptrdiff_t;
# 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef long unsigned int size_t;
# 328 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef int wchar_t;
# 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
typedef struct {
  long long __max_align_ll __attribute__((__aligned__(__alignof__(long long))));
  long double __max_align_ld __attribute__((__aligned__(__alignof__(long double))));
# 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4
} max_align_t;
# 233 "/usr/include/mpi.h" 2 3 4
# 265 "/usr/include/mpi.h" 3 4
# 1 "/usr/include/mpi_portable_platform.h" 1 3 4
# 266 "/usr/include/mpi.h" 2 3 4
# 342 "/usr/include/mpi.h" 3 4
typedef ptrdiff_t MPI_Aint;
typedef long long MPI_Offset;
typedef long long MPI_Count;
typedef struct ompi_communicator_t *MPI_Comm;
typedef struct ompi_datatype_t *MPI_Datatype;
typedef struct ompi_errhandler_t *MPI_Errhandler;
typedef struct ompi_file_t *MPI_File;
typedef struct ompi_group_t *MPI_Group;
typedef struct ompi_info_t *MPI_Info;
typedef struct ompi_op_t *MPI_Op;
typedef struct ompi_request_t *MPI_Request;
typedef struct ompi_message_t *MPI_Message;
typedef struct ompi_status_public_t MPI_Status;
typedef struct ompi_win_t *MPI_Win;
typedef struct mca_base_var_enum_t *MPI_T_enum;
typedef struct ompi_mpit_cvar_handle_t *MPI_T_cvar_handle;
typedef struct mca_base_pvar_handle_t *MPI_T_pvar_handle;
typedef struct mca_base_pvar_session_t *MPI_T_pvar_session;
struct ompi_status_public_t {
    int MPI_SOURCE;
    int MPI_TAG;
    int MPI_ERROR;
    int _cancelled;
    size_t _ucount;
};
typedef struct ompi_status_public_t ompi_status_public_t;
typedef int (MPI_Datarep_extent_function)(MPI_Datatype, MPI_Aint *, void *);
typedef int (MPI_Datarep_conversion_function)(void *, MPI_Datatype,
                                              int, void *, MPI_Offset, void *);
typedef void (MPI_Comm_errhandler_function)(MPI_Comm *, int *, ...);
typedef void (ompi_file_errhandler_fn)(MPI_File *, int *, ...);
typedef void (MPI_Win_errhandler_function)(MPI_Win *, int *, ...);
typedef void (MPI_User_function)(void *, void *, int *, MPI_Datatype *);
typedef int (MPI_Comm_copy_attr_function)(MPI_Comm, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Comm_delete_attr_function)(MPI_Comm, int, void *, void *);
typedef int (MPI_Type_copy_attr_function)(MPI_Datatype, int, void *,
                                            void *, void *, int *);
typedef int (MPI_Type_delete_attr_function)(MPI_Datatype, int,
                                              void *, void *);
typedef int (MPI_Win_copy_attr_function)(MPI_Win, int, void *,
                                           void *, void *, int *);
typedef int (MPI_Win_delete_attr_function)(MPI_Win, int, void *, void *);
typedef int (MPI_Grequest_query_function)(void *, MPI_Status *);
typedef int (MPI_Grequest_free_function)(void *);
typedef int (MPI_Grequest_cancel_function)(void *, int);
typedef MPI_Comm_errhandler_function MPI_Comm_errhandler_fn
        __attribute__((__deprecated__("MPI_Comm_errhandler_fn was deprecated in MPI-2.2; use MPI_Comm_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_fn
        __attribute__((__deprecated__("MPI_File_errhandler_fn was deprecated in MPI-2.2; use MPI_File_errhandler_function instead")));
typedef ompi_file_errhandler_fn MPI_File_errhandler_function;
typedef MPI_Win_errhandler_function MPI_Win_errhandler_fn
        __attribute__((__deprecated__("MPI_Win_errhandler_fn was deprecated in MPI-2.2; use MPI_Win_errhandler_function instead")));
# 509 "/usr/include/mpi.h" 3 4
enum {
    MPI_TAG_UB,
    MPI_HOST,
    MPI_IO,
    MPI_WTIME_IS_GLOBAL,
    MPI_APPNUM,
    MPI_LASTUSEDCODE,
    MPI_UNIVERSE_SIZE,
    MPI_WIN_BASE,
    MPI_WIN_SIZE,
    MPI_WIN_DISP_UNIT,
    MPI_WIN_CREATE_FLAVOR,
    MPI_WIN_MODEL,
    IMPI_CLIENT_SIZE,
    IMPI_CLIENT_COLOR,
    IMPI_HOST_SIZE,
    IMPI_HOST_COLOR
};
# 623 "/usr/include/mpi.h" 3 4
enum {
  MPI_IDENT,
  MPI_CONGRUENT,
  MPI_SIMILAR,
  MPI_UNEQUAL
};
enum {
  MPI_THREAD_SINGLE,
  MPI_THREAD_FUNNELED,
  MPI_THREAD_SERIALIZED,
  MPI_THREAD_MULTIPLE
};
enum {
  MPI_COMBINER_NAMED,
  MPI_COMBINER_DUP,
  MPI_COMBINER_CONTIGUOUS,
  MPI_COMBINER_VECTOR,
  MPI_COMBINER_HVECTOR_INTEGER,
  MPI_COMBINER_HVECTOR,
  MPI_COMBINER_INDEXED,
  MPI_COMBINER_HINDEXED_INTEGER,
  MPI_COMBINER_HINDEXED,
  MPI_COMBINER_INDEXED_BLOCK,
  MPI_COMBINER_STRUCT_INTEGER,
  MPI_COMBINER_STRUCT,
  MPI_COMBINER_SUBARRAY,
  MPI_COMBINER_DARRAY,
  MPI_COMBINER_F90_REAL,
  MPI_COMBINER_F90_COMPLEX,
  MPI_COMBINER_F90_INTEGER,
  MPI_COMBINER_RESIZED,
  MPI_COMBINER_HINDEXED_BLOCK
};
enum {
  MPI_COMM_TYPE_SHARED,
  OMPI_COMM_TYPE_HWTHREAD,
  OMPI_COMM_TYPE_CORE,
  OMPI_COMM_TYPE_L1CACHE,
  OMPI_COMM_TYPE_L2CACHE,
  OMPI_COMM_TYPE_L3CACHE,
  OMPI_COMM_TYPE_SOCKET,
  OMPI_COMM_TYPE_NUMA,
  OMPI_COMM_TYPE_BOARD,
  OMPI_COMM_TYPE_HOST,
  OMPI_COMM_TYPE_CU,
  OMPI_COMM_TYPE_CLUSTER
};
enum {
  MPI_T_VERBOSITY_USER_BASIC,
  MPI_T_VERBOSITY_USER_DETAIL,
  MPI_T_VERBOSITY_USER_ALL,
  MPI_T_VERBOSITY_TUNER_BASIC,
  MPI_T_VERBOSITY_TUNER_DETAIL,
  MPI_T_VERBOSITY_TUNER_ALL,
  MPI_T_VERBOSITY_MPIDEV_BASIC,
  MPI_T_VERBOSITY_MPIDEV_DETAIL,
  MPI_T_VERBOSITY_MPIDEV_ALL
};
enum {
  MPI_T_SCOPE_CONSTANT,
  MPI_T_SCOPE_READONLY,
  MPI_T_SCOPE_LOCAL,
  MPI_T_SCOPE_GROUP,
  MPI_T_SCOPE_GROUP_EQ,
  MPI_T_SCOPE_ALL,
  MPI_T_SCOPE_ALL_EQ
};
enum {
  MPI_T_BIND_NO_OBJECT,
  MPI_T_BIND_MPI_COMM,
  MPI_T_BIND_MPI_DATATYPE,
  MPI_T_BIND_MPI_ERRHANDLER,
  MPI_T_BIND_MPI_FILE,
  MPI_T_BIND_MPI_GROUP,
  MPI_T_BIND_MPI_OP,
  MPI_T_BIND_MPI_REQUEST,
  MPI_T_BIND_MPI_WIN,
  MPI_T_BIND_MPI_MESSAGE,
  MPI_T_BIND_MPI_INFO
};
enum {
  MPI_T_PVAR_CLASS_STATE,
  MPI_T_PVAR_CLASS_LEVEL,
  MPI_T_PVAR_CLASS_SIZE,
  MPI_T_PVAR_CLASS_PERCENTAGE,
  MPI_T_PVAR_CLASS_HIGHWATERMARK,
  MPI_T_PVAR_CLASS_LOWWATERMARK,
  MPI_T_PVAR_CLASS_COUNTER,
  MPI_T_PVAR_CLASS_AGGREGATE,
  MPI_T_PVAR_CLASS_TIMER,
  MPI_T_PVAR_CLASS_GENERIC
};
# 820 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_DELETE_FN( MPI_Datatype datatype,
                                                  int type_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_NULL_COPY_FN( MPI_Datatype datatype,
                                                int type_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_TYPE_DUP_FN( MPI_Datatype datatype,
                                          int type_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_DELETE_FN( MPI_Comm comm,
                                                  int comm_keyval,
                                                  void* attribute_val_out,
                                                  void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_NULL_COPY_FN( MPI_Comm comm,
                                                int comm_keyval,
                                                void* extra_state,
                                                void* attribute_val_in,
                                                void* attribute_val_out,
                                                int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_COMM_DUP_FN( MPI_Comm comm, int comm_keyval,
                                          void* extra_state,
                                          void* attribute_val_in,
                                          void* attribute_val_out,
                                          int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_DELETE_FN( MPI_Win window,
                                                 int win_keyval,
                                                 void* attribute_val_out,
                                                 void* extra_state );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_NULL_COPY_FN( MPI_Win window, int win_keyval,
                                               void* extra_state,
                                               void* attribute_val_in,
                                               void* attribute_val_out,
                                               int* flag );
__attribute__((visibility("default"))) int OMPI_C_MPI_WIN_DUP_FN( MPI_Win window, int win_keyval,
                                         void* extra_state,
                                         void* attribute_val_in,
                                         void* attribute_val_out,
                                         int* flag );
# 874 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_world;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_self;
__attribute__((visibility("default"))) extern struct ompi_predefined_communicator_t ompi_mpi_comm_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_empty;
__attribute__((visibility("default"))) extern struct ompi_predefined_group_t ompi_mpi_group_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_request_t ompi_request_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_message_t ompi_message_no_proc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_min;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_max;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_sum;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_prod;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_land;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_band;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_lxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_bxor;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_maxloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_minloc;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_replace;
__attribute__((visibility("default"))) extern struct ompi_predefined_op_t ompi_mpi_op_no_op;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_datatype_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_signed_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_char;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_byte;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_short;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_unsigned_long_long;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_double;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_wchar;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_packed;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cxx_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_character;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_ldblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2integer;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2real;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblprec;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2cplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_2dblcplex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_float_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_double_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_longdbl_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_short_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_long_int;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_logical8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer1;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_integer16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real2;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real4;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_real16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex8;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex16;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_complex32;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint8_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint16_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint32_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_int64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_uint64_t;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_aint;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_offset;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_count;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_bool;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_float_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_datatype_t ompi_mpi_c_long_double_complex;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errhandler_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_are_fatal;
__attribute__((visibility("default"))) extern struct ompi_predefined_errhandler_t ompi_mpi_errors_return;
__attribute__((visibility("default"))) extern struct ompi_predefined_win_t ompi_mpi_win_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_file_t ompi_mpi_file_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_null;
__attribute__((visibility("default"))) extern struct ompi_predefined_info_t ompi_mpi_info_env;
__attribute__((visibility("default"))) extern int *MPI_F_STATUS_IGNORE;
__attribute__((visibility("default"))) extern int *MPI_F_STATUSES_IGNORE;
# 1185 "/usr/include/mpi.h" 3 4
__attribute__((visibility("default"))) int MPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                  int target_rank, MPI_Aint target_disp, int target_count,
                                  MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int MPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int MPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int MPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, const int recvcounts[],
                                  const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                 void *baseptr);
__attribute__((visibility("default"))) int MPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                 MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                 MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                 const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                 void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                 MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                             int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                             int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int MPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int MPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                   const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int MPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                int periods[], int coords[]);
__attribute__((visibility("default"))) int MPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                const int periods[], int *newrank);
__attribute__((visibility("default"))) int MPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int MPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                  int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int MPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int MPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int MPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int MPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                   MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int MPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int MPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                          MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                          int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm MPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int MPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int MPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int MPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                         const int degrees[], const int targets[],
                                         const int weights[], MPI_Info info,
                                         int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int MPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                  int indegree, const int sources[],
                                                  const int sourceweights[],
                                                  int outdegree,
                                                  const int destinations[],
                                                  const int destweights[],
                                                  MPI_Info info, int reorder,
                                                  MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                           int sources[], int sourceweights[],
                                           int maxoutdegree,
                                           int destinations[],
                                           int destweights[]);
__attribute__((visibility("default"))) int MPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                  int *inneighbors,
                                                  int *outneighbors,
                                                  int *weighted);
__attribute__((visibility("default"))) int MPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int MPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int MPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int MPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int MPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int MPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int MPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                  int root, MPI_Comm comm, MPI_Comm *intercomm,
                                  int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                           const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                           int root, MPI_Comm comm, MPI_Comm *intercomm,
                                           int array_of_errcodes[]);
__attribute__((visibility("default"))) int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int MPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int MPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                        void *result_addr, MPI_Datatype datatype, int target_rank,
                                        MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int MPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler MPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int MPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int MPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int MPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                    int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File MPI_File_f2c(int file);
__attribute__((visibility("default"))) int MPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int MPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                 MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int MPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int MPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int MPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int MPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                     MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int MPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                     MPI_Datatype *etype,
                                     MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int MPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                    int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                        int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read(MPI_File fh, void *buf, int count,
                                 MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all(MPI_File fh, void *buf, int count,
                                     MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write(MPI_File fh, const void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all(MPI_File fh, const void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iread_all(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                            MPI_Offset *disp);
__attribute__((visibility("default"))) int MPI_File_read_shared(MPI_File fh, void *buf, int count,
                                        MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_shared(MPI_File fh, const void *buf, int count,
      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int MPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int MPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                              int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                           MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                               MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                            MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int MPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int MPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int MPI_Finalize(void);
__attribute__((visibility("default"))) int MPI_Finalized(int *flag);
__attribute__((visibility("default"))) int MPI_Free_mem(void *base);
__attribute__((visibility("default"))) int MPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                              void *recvbuf, int recvcount, MPI_Datatype recvtype,
                              int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, const int recvcounts[], const int displs[],
                               MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int MPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int MPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype, MPI_Count *count);
__attribute__((visibility("default"))) int MPI_Get(void *origin_addr, int origin_count,
                           MPI_Datatype origin_datatype, int target_rank,
                           MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                      void *result_addr, int result_count, MPI_Datatype result_datatype,
                                      int target_rank, MPI_Aint target_disp, int target_count,
                                      MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int MPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                    const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int MPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                 int index[], int edges[]);
__attribute__((visibility("default"))) int MPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                 int *newrank);
__attribute__((visibility("default"))) int MPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int MPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                       int neighbors[]);
__attribute__((visibility("default"))) int MPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int MPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int MPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                      MPI_Grequest_free_function *free_fn,
                                      MPI_Grequest_cancel_function *cancel_fn,
                                      void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int MPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int MPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group MPI_Group_f2c(int group);
__attribute__((visibility("default"))) int MPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                  MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                          MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                        MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int MPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int MPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                             MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int MPI_Group_union(MPI_Group group1, MPI_Group group2,
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int MPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Improbe(int source, int tag, MPI_Comm comm,
                               int *flag, MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Imrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int MPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int MPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info MPI_Info_f2c(int info);
__attribute__((visibility("default"))) int MPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int MPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                char *value, int *flag);
__attribute__((visibility("default"))) int MPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int MPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int MPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                         int *flag);
__attribute__((visibility("default"))) int MPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int MPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int MPI_Initialized(int *flag);
__attribute__((visibility("default"))) int MPI_Init_thread(int *argc, char ***argv, int required,
                                   int *provided);
__attribute__((visibility("default"))) int MPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                        MPI_Comm bridge_comm, int remote_leader,
                                        int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                       MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                              MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int MPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int MPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message MPI_Message_f2c(int message);
__attribute__((visibility("default"))) int MPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Mrecv(void *buf, int count, MPI_Datatype type,
                             MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int displs[],
                                           MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                         void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                         MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                          void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                          void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int MPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int MPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op MPI_Op_f2c(int op);
__attribute__((visibility("default"))) int MPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int MPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                     MPI_Datatype datatype, void *outbuf,
                                     MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int MPI_Pack_external_size(const char datarep[], int incount,
                                          MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int MPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                            void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                 int *size);
__attribute__((visibility("default"))) int MPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Publish_name(const char *service_name, MPI_Info info,
                                    const char *port_name);
__attribute__((visibility("default"))) int MPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                           int target_rank, MPI_Aint target_disp, int target_count,
                           MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int MPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                 int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                            int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                              MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                    MPI_Datatype datatype, MPI_Op op);
__attribute__((visibility("default"))) int MPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                      MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Register_datarep(const char *datarep,
                                        MPI_Datarep_conversion_function *read_conversion_fn,
                                        MPI_Datarep_conversion_function *write_conversion_fn,
                                        MPI_Datarep_extent_function *dtype_file_extent_fn,
                                        void *extra_state);
__attribute__((visibility("default"))) int MPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request MPI_Request_f2c(int request);
__attribute__((visibility("default"))) int MPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Request_get_status(MPI_Request request, int *flag,
                                          MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                            MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op,
                                       MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_cout,
                            MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                            MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                 int dest, int tag, MPI_Comm comm,
                                 MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                            int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                int dest, int sendtag, void *recvbuf, int recvcount,
                                MPI_Datatype recvtype, int source, int recvtag,
                                MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                        int dest, int sendtag, int source, int recvtag,
                                        MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int MPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int MPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int MPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int MPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int MPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                           int count);
__attribute__((visibility("default"))) int MPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                             MPI_Count count);
__attribute__((visibility("default"))) int MPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                               MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Testany(int count, MPI_Request array_of_requests[], int *index,
                               int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int MPI_Testsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int MPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                       MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_darray(int size, int rank, int ndims,
                                          const int gsize_array[], const int distrib_array[],
                                          const int darg_array[], const int psize_array[],
                                          int order, MPI_Datatype oldtype,
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed_block(int count, int blocklength,
                                                  const MPI_Aint array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                            const MPI_Aint array_of_displacements[],
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                           MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                          MPI_Type_delete_attr_function *type_delete_attr_fn,
                                          int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Type_create_indexed_block(int count, int blocklength,
                                                 const int array_of_displacements[],
                                                 MPI_Datatype oldtype,
                                                 MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                          const MPI_Aint array_of_displacements[],
                                          const MPI_Datatype array_of_types[],
                                          MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                            const int start_array[], int order,
                                            MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                           MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int MPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype MPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int MPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                         int max_addresses, int max_datatypes,
                                         int array_of_integers[],
                                         MPI_Aint array_of_addresses[],
                                         MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int MPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                         int *num_addresses, int *num_datatypes,
                                         int *combiner);
__attribute__((visibility("default"))) int MPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                       MPI_Aint *extent);
__attribute__((visibility("default"))) int MPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                         MPI_Count *extent);
__attribute__((visibility("default"))) int MPI_Type_get_name(MPI_Datatype type, char *type_name,
                                     int *resultlen);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                            MPI_Aint *true_extent);
__attribute__((visibility("default"))) int MPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                              MPI_Count *true_extent);
__attribute__((visibility("default"))) int MPI_Type_indexed(int count, const int array_of_blocklengths[],
                                    const int array_of_displacements[],
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int MPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                     void *attr_val);
__attribute__((visibility("default"))) int MPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int MPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int MPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int MPI_Type_vector(int count, int blocklength, int stride,
                                   MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int MPI_Unpack(const void *inbuf, int insize, int *position,
                              void *outbuf, int outcount, MPI_Datatype datatype,
                              MPI_Comm comm);
__attribute__((visibility("default"))) int MPI_Unpublish_name(const char *service_name, MPI_Info info, const char *port_name);
__attribute__((visibility("default"))) int MPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                        MPI_Aint *position, void *outbuf, int outcount,
                                        MPI_Datatype datatype);
__attribute__((visibility("default"))) int MPI_Waitall(int count, MPI_Request array_of_requests[],
                               MPI_Status *array_of_statuses);
__attribute__((visibility("default"))) int MPI_Waitany(int count, MPI_Request array_of_requests[],
                               int *index, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int MPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                int *outcount, int array_of_indices[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                    MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                           MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int MPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int MPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                  MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                             MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                         MPI_Win_delete_attr_function *win_delete_attr_fn,
                                         int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int MPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int MPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win MPI_Win_f2c(int win);
__attribute__((visibility("default"))) int MPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int MPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int MPI_Win_get_attr(MPI_Win win, int win_keyval,
                                    void *attribute_val, int *flag);
__attribute__((visibility("default"))) int MPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int MPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int MPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int MPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int MPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int MPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int MPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int MPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int MPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int MPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int MPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int MPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double MPI_Wtick(void);
__attribute__((visibility("default"))) double MPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_Abort(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                   int target_rank, MPI_Aint target_disp, int target_count,
                                   MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Add_error_class(int *errorclass);
__attribute__((visibility("default"))) int PMPI_Add_error_code(int errorclass, int *errorcode);
__attribute__((visibility("default"))) int PMPI_Add_error_string(int errorcode, const char *string);
__attribute__((visibility("default"))) int PMPI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                  void *recvbuf, int recvcount,
                                  MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                   void *recvbuf, const int recvcounts[],
                                   const int displs[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alloc_mem(MPI_Aint size, MPI_Info info,
                                  void *baseptr);
__attribute__((visibility("default"))) int PMPI_Allreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iallreduce(const void *sendbuf, void *recvbuf, int count,
                                  MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[],
                                  MPI_Datatype sendtype, void *recvbuf, const int recvcounts[],
                                  const int rdispls[], MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Alltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ialltoallw(const void *sendbuf, const int sendcounts[], const int sdispls[], const MPI_Datatype sendtypes[],
                                  void *recvbuf, const int recvcounts[], const int rdispls[], const MPI_Datatype recvtypes[],
                                  MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create(MPI_Comm comm_old, int n, const int nodes[],
                                          const int degrees[], const int targets[],
                                          const int weights[], MPI_Info info,
                                          int reorder, MPI_Comm * newcomm);
__attribute__((visibility("default"))) int PMPI_Dist_graph_create_adjacent(MPI_Comm comm_old,
                                                   int indegree, const int sources[],
                                                   const int sourceweights[],
                                                   int outdegree,
                                                   const int destinations[],
                                                   const int destweights[],
                                                   MPI_Info info, int reorder,
                                                   MPI_Comm *comm_dist_graph);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors(MPI_Comm comm, int maxindegree,
                                            int sources[], int sourceweights[],
                                            int maxoutdegree,
                                             int destinations[],
                                            int destweights[]);
__attribute__((visibility("default"))) int PMPI_Dist_graph_neighbors_count(MPI_Comm comm,
                                                   int *inneighbors,
                                                   int *outneighbors,
                                                   int *weighted);
__attribute__((visibility("default"))) int PMPI_Barrier(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibarrier(MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bcast(void *buffer, int count, MPI_Datatype datatype,
                              int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ibcast(void *buffer, int count, MPI_Datatype datatype,
                                  int root, MPI_Comm comm,
             MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Bsend(const void *buf, int count, MPI_Datatype datatype,
                              int dest, int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Bsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Buffer_attach(void *buffer, int size);
__attribute__((visibility("default"))) int PMPI_Buffer_detach(void *buffer, int *size);
__attribute__((visibility("default"))) int PMPI_Cancel(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_create(MPI_Comm old_comm, int ndims, const int dims[],
                                    const int periods[], int reorder, MPI_Comm *comm_cart);
__attribute__((visibility("default"))) int PMPI_Cart_get(MPI_Comm comm, int maxdims, int dims[],
                                 int periods[], int coords[]);
__attribute__((visibility("default"))) int PMPI_Cart_map(MPI_Comm comm, int ndims, const int dims[],
                                 const int periods[], int *newrank);
__attribute__((visibility("default"))) int PMPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank);
__attribute__((visibility("default"))) int PMPI_Cart_shift(MPI_Comm comm, int direction, int disp,
                                   int *rank_source, int *rank_dest);
__attribute__((visibility("default"))) int PMPI_Cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *new_comm);
__attribute__((visibility("default"))) int PMPI_Cartdim_get(MPI_Comm comm, int *ndims);
__attribute__((visibility("default"))) int PMPI_Close_port(const char *port_name);
__attribute__((visibility("default"))) int PMPI_Comm_accept(const char *port_name, MPI_Info info, int root,
                                    MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_c2f(MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Comm_call_errhandler(MPI_Comm comm, int errorcode);
__attribute__((visibility("default"))) int PMPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result);
__attribute__((visibility("default"))) int PMPI_Comm_connect(const char *port_name, MPI_Info info, int root,
                                     MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create_errhandler(MPI_Comm_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_create_keyval(MPI_Comm_copy_attr_function *comm_copy_attr_fn,
                                           MPI_Comm_delete_attr_function *comm_delete_attr_fn,
                                           int *comm_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_delete_attr(MPI_Comm comm, int comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_disconnect(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_idup(MPI_Comm comm, MPI_Comm *newcomm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Comm_dup_with_info(MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) MPI_Comm PMPI_Comm_f2c(int comm);
__attribute__((visibility("default"))) int PMPI_Comm_free_keyval(int *comm_keyval);
__attribute__((visibility("default"))) int PMPI_Comm_free(MPI_Comm *comm);
__attribute__((visibility("default"))) int PMPI_Comm_get_attr(MPI_Comm comm, int comm_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Comm_get_errhandler(MPI_Comm comm, MPI_Errhandler *erhandler);
__attribute__((visibility("default"))) int PMPI_Comm_get_info(MPI_Comm comm, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Comm_get_name(MPI_Comm comm, char *comm_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Comm_get_parent(MPI_Comm *parent);
__attribute__((visibility("default"))) int PMPI_Comm_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_join(int fd, MPI_Comm *intercomm);
__attribute__((visibility("default"))) int PMPI_Comm_rank(MPI_Comm comm, int *rank);
__attribute__((visibility("default"))) int PMPI_Comm_remote_group(MPI_Comm comm, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Comm_remote_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_set_attr(MPI_Comm comm, int comm_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Comm_set_errhandler(MPI_Comm comm, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Comm_set_info(MPI_Comm comm, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Comm_set_name(MPI_Comm comm, const char *comm_name);
__attribute__((visibility("default"))) int PMPI_Comm_size(MPI_Comm comm, int *size);
__attribute__((visibility("default"))) int PMPI_Comm_spawn(const char *command, char *argv[], int maxprocs, MPI_Info info,
                                   int root, MPI_Comm comm, MPI_Comm *intercomm,
                                   int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_spawn_multiple(int count, char *array_of_commands[], char **array_of_argv[],
                                            const int array_of_maxprocs[], const MPI_Info array_of_info[],
                                            int root, MPI_Comm comm, MPI_Comm *intercomm,
                                            int array_of_errcodes[]);
__attribute__((visibility("default"))) int PMPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_split_type(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
__attribute__((visibility("default"))) int PMPI_Comm_test_inter(MPI_Comm comm, int *flag);
__attribute__((visibility("default"))) int PMPI_Compare_and_swap(const void *origin_addr, const void *compare_addr,
                                         void *result_addr, MPI_Datatype datatype, int target_rank,
                                         MPI_Aint target_disp, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Dims_create(int nnodes, int ndims, int dims[]);
__attribute__((visibility("default"))) int PMPI_Errhandler_c2f(MPI_Errhandler errhandler);
__attribute__((visibility("default"))) MPI_Errhandler PMPI_Errhandler_f2c(int errhandler);
__attribute__((visibility("default"))) int PMPI_Errhandler_free(MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Error_class(int errorcode, int *errorclass);
__attribute__((visibility("default"))) int PMPI_Error_string(int errorcode, char *string, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Exscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Fetch_and_op(const void *origin_addr, void *result_addr, MPI_Datatype datatype,
                                     int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Iexscan(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_c2f(MPI_File file);
__attribute__((visibility("default"))) MPI_File PMPI_File_f2c(int file);
__attribute__((visibility("default"))) int PMPI_File_call_errhandler(MPI_File fh, int errorcode);
__attribute__((visibility("default"))) int PMPI_File_create_errhandler(MPI_File_errhandler_function *function,
                                               MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_set_errhandler( MPI_File file, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_File_get_errhandler( MPI_File file, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_File_open(MPI_Comm comm, const char *filename, int amode,
                                  MPI_Info info, MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_close(MPI_File *fh);
__attribute__((visibility("default"))) int PMPI_File_delete(const char *filename, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_set_size(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_preallocate(MPI_File fh, MPI_Offset size);
__attribute__((visibility("default"))) int PMPI_File_get_size(MPI_File fh, MPI_Offset *size);
__attribute__((visibility("default"))) int PMPI_File_get_group(MPI_File fh, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_File_get_amode(MPI_File fh, int *amode);
__attribute__((visibility("default"))) int PMPI_File_set_info(MPI_File fh, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_info(MPI_File fh, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype,
                                      MPI_Datatype filetype, const char *datarep, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_File_get_view(MPI_File fh, MPI_Offset *disp,
                                      MPI_Datatype *etype,
                                      MPI_Datatype *filetype, char *datarep);
__attribute__((visibility("default"))) int PMPI_File_read_at(MPI_File fh, MPI_Offset offset, void *buf,
                                     int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                         int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                      int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                          int count, MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_at(MPI_File fh, MPI_Offset offset, void *buf,
                                      int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at(MPI_File fh, MPI_Offset offset, const void *buf,
                                       int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_at_all(MPI_File fh, MPI_Offset offset, void *buf,
                                          int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_at_all(MPI_File fh, MPI_Offset offset, const void *buf,
                                           int count, MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read(MPI_File fh, void *buf, int count,
                                  MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all(MPI_File fh, void *buf, int count,
                                      MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write(MPI_File fh, const void *buf, int count,
                                   MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all(MPI_File fh, const void *buf, int count,
                                       MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread(MPI_File fh, void *buf, int count,
                                   MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite(MPI_File fh, const void *buf, int count,
                                    MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iread_all(MPI_File fh, void *buf, int count,
                                       MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_all(MPI_File fh, const void *buf, int count,
                                        MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_seek(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_get_byte_offset(MPI_File fh, MPI_Offset offset,
                                             MPI_Offset *disp);
__attribute__((visibility("default"))) int PMPI_File_read_shared(MPI_File fh, void *buf, int count,
                                         MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_shared(MPI_File fh, const void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_iread_shared(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_iwrite_shared(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_File_read_ordered(MPI_File fh, void *buf, int count,
                                          MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered(MPI_File fh, const void *buf, int count,
                                           MPI_Datatype datatype, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_seek_shared(MPI_File fh, MPI_Offset offset, int whence);
__attribute__((visibility("default"))) int PMPI_File_get_position_shared(MPI_File fh, MPI_Offset *offset);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_begin(MPI_File fh, MPI_Offset offset, void *buf,
                                               int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_at_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_begin(MPI_File fh, MPI_Offset offset, const void *buf,
                                                int count, MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_at_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_all_begin(MPI_File fh, void *buf, int count,
                                            MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_all_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_all_begin(MPI_File fh, const void *buf, int count,
                                             MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_all_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_begin(MPI_File fh, void *buf, int count,
                                                MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_read_ordered_end(MPI_File fh, void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_begin(MPI_File fh, const void *buf, int count,
                                                 MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_File_write_ordered_end(MPI_File fh, const void *buf, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_File_get_type_extent(MPI_File fh, MPI_Datatype datatype,
                                             MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_File_set_atomicity(MPI_File fh, int flag);
__attribute__((visibility("default"))) int PMPI_File_get_atomicity(MPI_File fh, int *flag);
__attribute__((visibility("default"))) int PMPI_File_sync(MPI_File fh);
__attribute__((visibility("default"))) int PMPI_Finalize(void);
__attribute__((visibility("default"))) int PMPI_Finalized(int *flag);
__attribute__((visibility("default"))) int PMPI_Free_mem(void *base);
__attribute__((visibility("default"))) int PMPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                               void *recvbuf, int recvcount, MPI_Datatype recvtype,
                               int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, const int recvcounts[], const int displs[],
                                MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Get_address(const void *location, MPI_Aint *address);
__attribute__((visibility("default"))) int PMPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements(const MPI_Status *status, MPI_Datatype datatype,
                                     int *count);
__attribute__((visibility("default"))) int PMPI_Get_elements_x(const MPI_Status *status, MPI_Datatype datatype,
                                       MPI_Count *count);
__attribute__((visibility("default"))) int PMPI_Get(void *origin_addr, int origin_count,
                            MPI_Datatype origin_datatype, int target_rank,
                            MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                       void *result_addr, int result_count, MPI_Datatype result_datatype,
                                       int target_rank, MPI_Aint target_disp, int target_count,
                                       MPI_Datatype target_datatype, MPI_Op op, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Get_library_version(char *version, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_processor_name(char *name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Get_version(int *version, int *subversion);
__attribute__((visibility("default"))) int PMPI_Graph_create(MPI_Comm comm_old, int nnodes, const int index[],
                                     const int edges[], int reorder, MPI_Comm *comm_graph);
__attribute__((visibility("default"))) int PMPI_Graph_get(MPI_Comm comm, int maxindex, int maxedges,
                                  int index[], int edges[]);
__attribute__((visibility("default"))) int PMPI_Graph_map(MPI_Comm comm, int nnodes, const int index[], const int edges[],
                                  int *newrank);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors_count(MPI_Comm comm, int rank, int *nneighbors);
__attribute__((visibility("default"))) int PMPI_Graph_neighbors(MPI_Comm comm, int rank, int maxneighbors,
                                        int neighbors[]);
__attribute__((visibility("default"))) int PMPI_Graphdims_get(MPI_Comm comm, int *nnodes, int *nedges);
__attribute__((visibility("default"))) int PMPI_Grequest_complete(MPI_Request request);
__attribute__((visibility("default"))) int PMPI_Grequest_start(MPI_Grequest_query_function *query_fn,
                                       MPI_Grequest_free_function *free_fn,
                                       MPI_Grequest_cancel_function *cancel_fn,
                                       void *extra_state, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Group_c2f(MPI_Group group);
__attribute__((visibility("default"))) int PMPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result);
__attribute__((visibility("default"))) int PMPI_Group_difference(MPI_Group group1, MPI_Group group2,
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_excl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) MPI_Group PMPI_Group_f2c(int group);
__attribute__((visibility("default"))) int PMPI_Group_free(MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Group_incl(MPI_Group group, int n, const int ranks[],
                                   MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_intersection(MPI_Group group1, MPI_Group group2,
                                           MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_excl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_range_incl(MPI_Group group, int n, int ranges[][3],
                                         MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Group_rank(MPI_Group group, int *rank);
__attribute__((visibility("default"))) int PMPI_Group_size(MPI_Group group, int *size);
__attribute__((visibility("default"))) int PMPI_Group_translate_ranks(MPI_Group group1, int n, const int ranks1[],
                                              MPI_Group group2, int ranks2[]);
__attribute__((visibility("default"))) int PMPI_Group_union(MPI_Group group1, MPI_Group group2,
                                    MPI_Group *newgroup);
__attribute__((visibility("default"))) int PMPI_Ibsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Improbe(int source, int tag, MPI_Comm comm,
                                int *flag, MPI_Message *message,
                                MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Imrecv(void *buf, int count, MPI_Datatype type,
                               MPI_Message *message, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Info_c2f(MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Info_create(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_delete(MPI_Info info, const char *key);
__attribute__((visibility("default"))) int PMPI_Info_dup(MPI_Info info, MPI_Info *newinfo);
__attribute__((visibility("default"))) MPI_Info PMPI_Info_f2c(int info);
__attribute__((visibility("default"))) int PMPI_Info_free(MPI_Info *info);
__attribute__((visibility("default"))) int PMPI_Info_get(MPI_Info info, const char *key, int valuelen,
                                 char *value, int *flag);
__attribute__((visibility("default"))) int PMPI_Info_get_nkeys(MPI_Info info, int *nkeys);
__attribute__((visibility("default"))) int PMPI_Info_get_nthkey(MPI_Info info, int n, char *key);
__attribute__((visibility("default"))) int PMPI_Info_get_valuelen(MPI_Info info, const char *key, int *valuelen,
                                          int *flag);
__attribute__((visibility("default"))) int PMPI_Info_set(MPI_Info info, const char *key, const char *value);
__attribute__((visibility("default"))) int PMPI_Init(int *argc, char ***argv);
__attribute__((visibility("default"))) int PMPI_Initialized(int *flag);
__attribute__((visibility("default"))) int PMPI_Init_thread(int *argc, char ***argv, int required,
                                    int *provided);
__attribute__((visibility("default"))) int PMPI_Intercomm_create(MPI_Comm local_comm, int local_leader,
                                         MPI_Comm bridge_comm, int remote_leader,
                                         int tag, MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Intercomm_merge(MPI_Comm intercomm, int high,
                                        MPI_Comm *newintercomm);
__attribute__((visibility("default"))) int PMPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Irsend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Isend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Issend(const void *buf, int count, MPI_Datatype datatype, int dest,
                               int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Is_thread_main(int *flag);
__attribute__((visibility("default"))) int PMPI_Lookup_name(const char *service_name, MPI_Info info, char *port_name);
__attribute__((visibility("default"))) int PMPI_Message_c2f(MPI_Message message);
__attribute__((visibility("default"))) MPI_Message PMPI_Message_f2c(int message);
__attribute__((visibility("default"))) int PMPI_Mprobe(int source, int tag, MPI_Comm comm,
                               MPI_Message *message,
                               MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Mrecv(void *buf, int count, MPI_Datatype type,
                              MPI_Message *message, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int displs[],
                                            MPI_Datatype recvtype, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                             void *recvbuf, const int recvcounts[], const int displs[],
                                             MPI_Datatype recvtype, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                          void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                          MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                           void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                           MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                           void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], const int sdispls[], MPI_Datatype sendtype,
                                            void *recvbuf, const int recvcounts[], const int rdispls[], MPI_Datatype recvtype,
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Neighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                           void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                           MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], const MPI_Aint sdispls[], const MPI_Datatype sendtypes[],
                                            void *recvbuf, const int recvcounts[], const MPI_Aint rdispls[], const MPI_Datatype recvtypes[],
                                            MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Op_c2f(MPI_Op op);
__attribute__((visibility("default"))) int PMPI_Op_commutative(MPI_Op op, int *commute);
__attribute__((visibility("default"))) int PMPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Open_port(MPI_Info info, char *port_name);
__attribute__((visibility("default"))) MPI_Op PMPI_Op_f2c(int op);
__attribute__((visibility("default"))) int PMPI_Op_free(MPI_Op *op);
__attribute__((visibility("default"))) int PMPI_Pack_external(const char datarep[], const void *inbuf, int incount,
                                      MPI_Datatype datatype, void *outbuf,
                                      MPI_Aint outsize, MPI_Aint *position);
__attribute__((visibility("default"))) int PMPI_Pack_external_size(const char datarep[], int incount,
                                           MPI_Datatype datatype, MPI_Aint *size);
__attribute__((visibility("default"))) int PMPI_Pack(const void *inbuf, int incount, MPI_Datatype datatype,
                             void *outbuf, int outsize, int *position, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm,
                                  int *size);
__attribute__((visibility("default"))) int PMPI_Pcontrol(const int level, ...);
__attribute__((visibility("default"))) int PMPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Publish_name(const char *service_name, MPI_Info info,
                                     const char *port_name);
__attribute__((visibility("default"))) int PMPI_Put(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                            int target_rank, MPI_Aint target_disp, int target_count,
                            MPI_Datatype target_datatype, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Query_thread(int *provided);
__attribute__((visibility("default"))) int PMPI_Raccumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                    int target_rank, MPI_Aint target_disp, int target_count,
                                    MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int source,
                                  int tag, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Recv(void *buf, int count, MPI_Datatype datatype, int source,
                             int tag, MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Reduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce(const void *sendbuf, void *recvbuf, int count,
                               MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_local(const void *inbuf, void *inoutbuf, int count,
                                     MPI_Datatype datatype, MPI_Op);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                       MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int recvcounts[],
                                        MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount,
                                              MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Register_datarep(const char *datarep,
                                         MPI_Datarep_conversion_function *read_conversion_fn,
                                         MPI_Datarep_conversion_function *write_conversion_fn,
                                         MPI_Datarep_extent_function *dtype_file_extent_fn,
                                         void *extra_state);
__attribute__((visibility("default"))) int PMPI_Request_c2f(MPI_Request request);
__attribute__((visibility("default"))) MPI_Request PMPI_Request_f2c(int request);
__attribute__((visibility("default"))) int PMPI_Request_free(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Request_get_status(MPI_Request request, int *flag,
                                           MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Rget(void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
                             MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rget_accumulate(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                                        void *result_addr, int result_count, MPI_Datatype result_datatype,
                                        int target_rank, MPI_Aint target_disp, int target_count,
                                        MPI_Datatype target_datatype, MPI_Op op,
                                        MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rput(const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
                             int target_rank, MPI_Aint target_disp, int target_cout,
                             MPI_Datatype target_datatype, MPI_Win win, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Rsend(const void *ibuf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Rsend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscan(const void *sendbuf, void *recvbuf, int count,
                             MPI_Datatype datatype, MPI_Op op, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                void *recvbuf, int recvcount, MPI_Datatype recvtype,
                                int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Scatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Iscatterv(const void *sendbuf, const int sendcounts[], const int displs[],
                                 MPI_Datatype sendtype, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int root, MPI_Comm comm, MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send_init(const void *buf, int count, MPI_Datatype datatype,
                                  int dest, int tag, MPI_Comm comm,
                                  MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest,
                             int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Sendrecv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
                                 int dest, int sendtag, void *recvbuf, int recvcount,
                                 MPI_Datatype recvtype, int source, int recvtag,
                                 MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Sendrecv_replace(void * buf, int count, MPI_Datatype datatype,
                                         int dest, int sendtag, int source, int recvtag,
                                         MPI_Comm comm, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Ssend_init(const void *buf, int count, MPI_Datatype datatype,
                                   int dest, int tag, MPI_Comm comm,
                                   MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest,
                              int tag, MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Start(MPI_Request *request);
__attribute__((visibility("default"))) int PMPI_Startall(int count, MPI_Request array_of_requests[]);
__attribute__((visibility("default"))) int PMPI_Status_c2f(const MPI_Status *c_status, int *f_status);
__attribute__((visibility("default"))) int PMPI_Status_f2c(const int *f_status, MPI_Status *c_status);
__attribute__((visibility("default"))) int PMPI_Status_set_cancelled(MPI_Status *status, int flag);
__attribute__((visibility("default"))) int PMPI_Status_set_elements(MPI_Status *status, MPI_Datatype datatype,
                                            int count);
__attribute__((visibility("default"))) int PMPI_Status_set_elements_x(MPI_Status *status, MPI_Datatype datatype,
                                              MPI_Count count);
__attribute__((visibility("default"))) int PMPI_Testall(int count, MPI_Request array_of_requests[], int *flag,
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Testany(int count, MPI_Request array_of_requests[], int *index, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test(MPI_Request *request, int *flag, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Test_cancelled(const MPI_Status *status, int *flag);
__attribute__((visibility("default"))) int PMPI_Testsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Topo_test(MPI_Comm comm, int *status);
__attribute__((visibility("default"))) int PMPI_Type_c2f(MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Type_commit(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_contiguous(int count, MPI_Datatype oldtype,
                                        MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_darray(int size, int rank, int ndims,
                                           const int gsize_array[], const int distrib_array[],
                                           const int darg_array[], const int psize_array[],
                                           int order, MPI_Datatype oldtype,
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_complex(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_integer(int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_f90_real(int p, int r, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed(int count, const int array_of_blocklengths[],
                                             const MPI_Aint array_of_displacements[],
                                             MPI_Datatype oldtype,
                                             MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride,
                                            MPI_Datatype oldtype,
                                            MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_keyval(MPI_Type_copy_attr_function *type_copy_attr_fn,
                                           MPI_Type_delete_attr_function *type_delete_attr_fn,
                                           int *type_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Type_create_hindexed_block(int count, int blocklength,
                                                   const MPI_Aint array_of_displacements[],
                                                   MPI_Datatype oldtype,
                                                   MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_indexed_block(int count, int blocklength,
                                                  const int array_of_displacements[],
                                                  MPI_Datatype oldtype,
                                                  MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_struct(int count, const int array_of_block_lengths[],
                                           const MPI_Aint array_of_displacements[],
                                           const MPI_Datatype array_of_types[],
                                           MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_subarray(int ndims, const int size_array[], const int subsize_array[],
                                             const int start_array[], int order,
                                             MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb,
                                            MPI_Aint extent, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_delete_attr(MPI_Datatype type, int type_keyval);
__attribute__((visibility("default"))) int PMPI_Type_dup(MPI_Datatype type, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_free(MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_free_keyval(int *type_keyval);
__attribute__((visibility("default"))) MPI_Datatype PMPI_Type_f2c(int datatype);
__attribute__((visibility("default"))) int PMPI_Type_get_attr(MPI_Datatype type, int type_keyval,
                                      void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Type_get_contents(MPI_Datatype mtype, int max_integers,
                                          int max_addresses, int max_datatypes,
                                          int array_of_integers[],
                                          MPI_Aint array_of_addresses[],
                                          MPI_Datatype array_of_datatypes[]);
__attribute__((visibility("default"))) int PMPI_Type_get_envelope(MPI_Datatype type, int *num_integers,
                                          int *num_addresses, int *num_datatypes,
                                          int *combiner);
__attribute__((visibility("default"))) int PMPI_Type_get_extent(MPI_Datatype type, MPI_Aint *lb,
                                        MPI_Aint *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_extent_x(MPI_Datatype type, MPI_Count *lb,
                                          MPI_Count *extent);
__attribute__((visibility("default"))) int PMPI_Type_get_name(MPI_Datatype type, char *type_name,
                                      int *resultlen);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint *true_lb,
                                             MPI_Aint *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_get_true_extent_x(MPI_Datatype datatype, MPI_Count *true_lb,
                                               MPI_Count *true_extent);
__attribute__((visibility("default"))) int PMPI_Type_indexed(int count, const int array_of_blocklengths[],
                                     const int array_of_displacements[],
                                     MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Type_match_size(int typeclass, int size, MPI_Datatype *type);
__attribute__((visibility("default"))) int PMPI_Type_set_attr(MPI_Datatype type, int type_keyval,
                                      void *attr_val);
__attribute__((visibility("default"))) int PMPI_Type_set_name(MPI_Datatype type, const char *type_name);
__attribute__((visibility("default"))) int PMPI_Type_size(MPI_Datatype type, int *size);
__attribute__((visibility("default"))) int PMPI_Type_size_x(MPI_Datatype type, MPI_Count *size);
__attribute__((visibility("default"))) int PMPI_Type_vector(int count, int blocklength, int stride,
                                    MPI_Datatype oldtype, MPI_Datatype *newtype);
__attribute__((visibility("default"))) int PMPI_Unpack(const void *inbuf, int insize, int *position,
                               void *outbuf, int outcount, MPI_Datatype datatype,
                               MPI_Comm comm);
__attribute__((visibility("default"))) int PMPI_Unpublish_name(const char *service_name, MPI_Info info,
                                       const char *port_name);
__attribute__((visibility("default"))) int PMPI_Unpack_external (const char datarep[], const void *inbuf, MPI_Aint insize,
                                         MPI_Aint *position, void *outbuf, int outcount,
                                         MPI_Datatype datatype);
__attribute__((visibility("default"))) int PMPI_Waitall(int count, MPI_Request array_of_requests[],
                                MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Waitany(int count, MPI_Request array_of_requests[],
                                int *index, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Wait(MPI_Request *request, MPI_Status *status);
__attribute__((visibility("default"))) int PMPI_Waitsome(int incount, MPI_Request array_of_requests[],
                                 int *outcount, int array_of_indices[],
                                 MPI_Status array_of_statuses[]);
__attribute__((visibility("default"))) int PMPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
                                     MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_allocate_shared(MPI_Aint size, int disp_unit, MPI_Info info,
                                            MPI_Comm comm, void *baseptr, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_attach(MPI_Win win, void *base, MPI_Aint size);
__attribute__((visibility("default"))) int PMPI_Win_c2f(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_call_errhandler(MPI_Win win, int errorcode);
__attribute__((visibility("default"))) int PMPI_Win_complete(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                                   MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm, MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_create_errhandler(MPI_Win_errhandler_function *function,
                                              MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_create_keyval(MPI_Win_copy_attr_function *win_copy_attr_fn,
                                          MPI_Win_delete_attr_function *win_delete_attr_fn,
                                          int *win_keyval, void *extra_state);
__attribute__((visibility("default"))) int PMPI_Win_delete_attr(MPI_Win win, int win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_detach(MPI_Win win, const void *base);
__attribute__((visibility("default"))) MPI_Win PMPI_Win_f2c(int win);
__attribute__((visibility("default"))) int PMPI_Win_fence(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_flush_local_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_free(MPI_Win *win);
__attribute__((visibility("default"))) int PMPI_Win_free_keyval(int *win_keyval);
__attribute__((visibility("default"))) int PMPI_Win_get_attr(MPI_Win win, int win_keyval,
                                     void *attribute_val, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_get_errhandler(MPI_Win win, MPI_Errhandler *errhandler);
__attribute__((visibility("default"))) int PMPI_Win_get_group(MPI_Win win, MPI_Group *group);
__attribute__((visibility("default"))) int PMPI_Win_get_info(MPI_Win win, MPI_Info *info_used);
__attribute__((visibility("default"))) int PMPI_Win_get_name(MPI_Win win, char *win_name, int *resultlen);
__attribute__((visibility("default"))) int PMPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_lock_all(int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_post(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_set_attr(MPI_Win win, int win_keyval, void *attribute_val);
__attribute__((visibility("default"))) int PMPI_Win_set_errhandler(MPI_Win win, MPI_Errhandler errhandler);
__attribute__((visibility("default"))) int PMPI_Win_set_info(MPI_Win win, MPI_Info info);
__attribute__((visibility("default"))) int PMPI_Win_set_name(MPI_Win win, const char *win_name);
__attribute__((visibility("default"))) int PMPI_Win_shared_query(MPI_Win win, int rank, MPI_Aint *size, int *disp_unit, void *baseptr);
__attribute__((visibility("default"))) int PMPI_Win_start(MPI_Group group, int assert, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_sync(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_test(MPI_Win win, int *flag);
__attribute__((visibility("default"))) int PMPI_Win_unlock(int rank, MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_unlock_all(MPI_Win win);
__attribute__((visibility("default"))) int PMPI_Win_wait(MPI_Win win);
__attribute__((visibility("default"))) double PMPI_Wtick(void);
__attribute__((visibility("default"))) double PMPI_Wtime(void);
__attribute__((visibility("default"))) int PMPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int PMPI_T_finalize (void);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                         int *verbosity, MPI_Datatype *datatype,
                                         MPI_T_enum *enumtype, char *desc,
                                         int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int PMPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int PMPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int PMPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int PMPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                            char *desc, int *desc_len, int *num_cvars,
                                            int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int PMPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int PMPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int PMPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                        int *verbosity, int *var_class, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                        int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int PMPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                            void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int PMPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                     const void *buf);
__attribute__((visibility("default"))) int PMPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int PMPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                         void *buf);
__attribute__((visibility("default"))) int PMPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int PMPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                        int *name_len);
__attribute__((visibility("default"))) int MPI_T_init_thread (int required, int *provided);
__attribute__((visibility("default"))) int MPI_T_finalize (void);
__attribute__((visibility("default"))) int MPI_T_cvar_get_num (int *num_cvar);
__attribute__((visibility("default"))) int MPI_T_cvar_get_info (int cvar_index, char *name, int *name_len,
                                        int *verbosity, MPI_Datatype *datatype,
                                        MPI_T_enum *enumtype, char *desc,
                                        int *desc_len, int *bind, int *scope);
__attribute__((visibility("default"))) int MPI_T_cvar_get_index (const char *name, int *cvar_index);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_alloc (int cvar_index, void *obj_handle,
                                            MPI_T_cvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_cvar_handle_free (MPI_T_cvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_cvar_read (MPI_T_cvar_handle handle, void *buf);
__attribute__((visibility("default"))) int MPI_T_cvar_write (MPI_T_cvar_handle handle, const void *buf);
__attribute__((visibility("default"))) int MPI_T_category_get_num(int *num_cat);
__attribute__((visibility("default"))) int MPI_T_category_get_info(int cat_index, char *name, int *name_len,
                                           char *desc, int *desc_len, int *num_cvars,
                                           int *num_pvars, int *num_categories);
__attribute__((visibility("default"))) int MPI_T_category_get_index (const char *name, int *category_index);
__attribute__((visibility("default"))) int MPI_T_category_get_cvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_pvars(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_get_categories(int cat_index, int len, int indices[]);
__attribute__((visibility("default"))) int MPI_T_category_changed(int *stamp);
__attribute__((visibility("default"))) int MPI_T_pvar_get_num(int *num_pvar);
__attribute__((visibility("default"))) int MPI_T_pvar_get_info(int pvar_index, char *name, int *name_len,
                                       int *verbosity, int *var_class, MPI_Datatype *datatype,
                                       MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
                                       int *readonly, int *continuous, int *atomic);
__attribute__((visibility("default"))) int MPI_T_pvar_get_index (const char *name, int var_class, int *pvar_index);
__attribute__((visibility("default"))) int MPI_T_pvar_session_create(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_session_free(MPI_T_pvar_session *session);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_alloc(MPI_T_pvar_session session, int pvar_index,
                                           void *obj_handle, MPI_T_pvar_handle *handle, int *count);
__attribute__((visibility("default"))) int MPI_T_pvar_handle_free(MPI_T_pvar_session session, MPI_T_pvar_handle *handle);
__attribute__((visibility("default"))) int MPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_read(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                   void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_write(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                    const void *buf);
__attribute__((visibility("default"))) int MPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
__attribute__((visibility("default"))) int MPI_T_pvar_readreset(MPI_T_pvar_session session, MPI_T_pvar_handle handle,
                                        void *buf);
__attribute__((visibility("default"))) int MPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *name, int *name_len);
__attribute__((visibility("default"))) int MPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
                                       int *name_len);
__attribute__((visibility("default"))) int MPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("MPI_Attr_delete was deprecated in MPI-2.0; use MPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_delete(MPI_Comm comm, int keyval)
            __attribute__((__deprecated__("PMPI_Attr_delete was deprecated in MPI-2.0; use PMPI_Comm_delete_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("MPI_Attr_get was deprecated in MPI-2.0; use MPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_get(MPI_Comm comm, int keyval, void *attribute_val, int *flag)
            __attribute__((__deprecated__("PMPI_Attr_get was deprecated in MPI-2.0; use PMPI_Comm_get_attr instead")));
__attribute__((visibility("default"))) int MPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("MPI_Attr_put was deprecated in MPI-2.0; use MPI_Comm_set_attr instead")));
__attribute__((visibility("default"))) int PMPI_Attr_put(MPI_Comm comm, int keyval, void *attribute_val)
            __attribute__((__deprecated__("PMPI_Attr_put was deprecated in MPI-2.0; use PMPI_Comm_set_attr instead")));
typedef int (MPI_Copy_function)(MPI_Comm, int, void *,
                                void *, void *, int *);
typedef int (MPI_Delete_function)(MPI_Comm, int, void *, void *);
__attribute__((visibility("default"))) int MPI_Keyval_create(MPI_Copy_function *copy_fn,
                                     MPI_Delete_function *delete_fn,
                                     int *keyval, void *extra_state)
            __attribute__((__deprecated__("MPI_Keyval_create was deprecated in MPI-2.0; use MPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_create(MPI_Copy_function *copy_fn,
                                      MPI_Delete_function *delete_fn,
                                      int *keyval, void *extra_state)
            __attribute__((__deprecated__("PMPI_Keyval_create was deprecated in MPI-2.0; use PMPI_Comm_create_keyval instead.")));
__attribute__((visibility("default"))) int MPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("MPI_Keyval_free was deprecated in MPI-2.0; MPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int PMPI_Keyval_free(int *keyval)
            __attribute__((__deprecated__("PMPI_Keyval_free was deprecated in MPI-2.0; PMPI_Comm_free_keyval instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_DUP_FN( MPI_Comm comm, int comm_keyval,
                                     void* extra_state,
                                     void* attribute_val_in,
                                     void* attribute_val_out,
                                     int* flag )
            __attribute__((__deprecated__("MPI_DUP_FN was deprecated in MPI-2.0; use MPI_COMM_DUP_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_COPY_FN( MPI_Comm comm, int comm_keyval,
                                           void* extra_state,
                                           void* attribute_val_in,
                                           void* attribute_val_out,
                                           int* flag )
            __attribute__((__deprecated__("MPI_NULL_COPY_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_COPY_FN instead.")));
__attribute__((visibility("default"))) int OMPI_C_MPI_NULL_DELETE_FN( MPI_Comm comm, int comm_keyval,
                                             void* attribute_val_out,
                                             void* extra_state )
            __attribute__((__deprecated__("MPI_NULL_DELETE_FN was deprecated in MPI-2.0; use MPI_COMM_NULL_DELETE_FN instead.")));
# 18 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h" 2
# 1 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h" 1
# 94 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
# 1 "/usr/include/inttypes.h" 1 3 4
# 25 "/usr/include/inttypes.h" 3 4
# 1 "/usr/include/features.h" 1 3 4
# 428 "/usr/include/features.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 442 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 443 "/usr/include/sys/cdefs.h" 2 3 4
# 1 "/usr/include/bits/long-double.h" 1 3 4
# 444 "/usr/include/sys/cdefs.h" 2 3 4
# 429 "/usr/include/features.h" 2 3 4
# 452 "/usr/include/features.h" 3 4
# 1 "/usr/include/gnu/stubs.h" 1 3 4
# 10 "/usr/include/gnu/stubs.h" 3 4
# 1 "/usr/include/gnu/stubs-64.h" 1 3 4
# 11 "/usr/include/gnu/stubs.h" 2 3 4
# 453 "/usr/include/features.h" 2 3 4
# 26 "/usr/include/inttypes.h" 2 3 4
# 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 1 3 4
# 9 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 26 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/bits/libc-header-start.h" 1 3 4
# 27 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/types.h" 1 3 4
# 27 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 28 "/usr/include/bits/types.h" 2 3 4
typedef unsigned char __u_char;
typedef unsigned short int __u_short;
typedef unsigned int __u_int;
typedef unsigned long int __u_long;
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef signed short int __int16_t;
typedef unsigned short int __uint16_t;
typedef signed int __int32_t;
typedef unsigned int __uint32_t;
typedef signed long int __int64_t;
typedef unsigned long int __uint64_t;
typedef __int8_t __int_least8_t;
typedef __uint8_t __uint_least8_t;
typedef __int16_t __int_least16_t;
typedef __uint16_t __uint_least16_t;
typedef __int32_t __int_least32_t;
typedef __uint32_t __uint_least32_t;
typedef __int64_t __int_least64_t;
typedef __uint64_t __uint_least64_t;
typedef long int __quad_t;
typedef unsigned long int __u_quad_t;
typedef long int __intmax_t;
typedef unsigned long int __uintmax_t;
# 140 "/usr/include/bits/types.h" 3 4
# 1 "/usr/include/bits/typesizes.h" 1 3 4
# 141 "/usr/include/bits/types.h" 2 3 4
typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;
typedef int __pid_t;
typedef struct { int __val[2]; } __fsid_t;
typedef long int __clock_t;
typedef unsigned long int __rlim_t;
typedef unsigned long int __rlim64_t;
typedef unsigned int __id_t;
typedef long int __time_t;
typedef unsigned int __useconds_t;
typedef long int __suseconds_t;
typedef int __daddr_t;
typedef int __key_t;
typedef int __clockid_t;
typedef void * __timer_t;
typedef long int __blksize_t;
typedef long int __blkcnt_t;
typedef long int __blkcnt64_t;
typedef unsigned long int __fsblkcnt_t;
typedef unsigned long int __fsblkcnt64_t;
typedef unsigned long int __fsfilcnt_t;
typedef unsigned long int __fsfilcnt64_t;
typedef long int __fsword_t;
typedef long int __ssize_t;
typedef long int __syscall_slong_t;
typedef unsigned long int __syscall_ulong_t;
typedef __off64_t __loff_t;
typedef char *__caddr_t;
typedef long int __intptr_t;
typedef unsigned int __socklen_t;
typedef int __sig_atomic_t;
# 28 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wchar.h" 1 3 4
# 29 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/wordsize.h" 1 3 4
# 30 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-intn.h" 1 3 4
# 24 "/usr/include/bits/stdint-intn.h" 3 4
typedef __int8_t int8_t;
typedef __int16_t int16_t;
typedef __int32_t int32_t;
typedef __int64_t int64_t;
# 35 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/bits/stdint-uintn.h" 1 3 4
# 24 "/usr/include/bits/stdint-uintn.h" 3 4
typedef __uint8_t uint8_t;
typedef __uint16_t uint16_t;
typedef __uint32_t uint32_t;
typedef __uint64_t uint64_t;
# 38 "/usr/include/stdint.h" 2 3 4
typedef __int_least8_t int_least8_t;
typedef __int_least16_t int_least16_t;
typedef __int_least32_t int_least32_t;
typedef __int_least64_t int_least64_t;
typedef __uint_least8_t uint_least8_t;
typedef __uint_least16_t uint_least16_t;
typedef __uint_least32_t uint_least32_t;
typedef __uint_least64_t uint_least64_t;
typedef signed char int_fast8_t;
typedef long int int_fast16_t;
typedef long int int_fast32_t;
typedef long int int_fast64_t;
# 71 "/usr/include/stdint.h" 3 4
typedef unsigned char uint_fast8_t;
typedef unsigned long int uint_fast16_t;
typedef unsigned long int uint_fast32_t;
typedef unsigned long int uint_fast64_t;
# 87 "/usr/include/stdint.h" 3 4
typedef long int intptr_t;
typedef unsigned long int uintptr_t;
# 101 "/usr/include/stdint.h" 3 4
typedef __intmax_t intmax_t;
typedef __uintmax_t uintmax_t;
# 10 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stdint.h" 2 3 4
# 28 "/usr/include/inttypes.h" 2 3 4
typedef int __gwchar_t;
# 266 "/usr/include/inttypes.h" 3 4
typedef struct
  {
    long int quot;
    long int rem;
  } imaxdiv_t;
# 290 "/usr/include/inttypes.h" 3 4
extern intmax_t imaxabs (intmax_t __n) __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern imaxdiv_t imaxdiv (intmax_t __numer, intmax_t __denom)
      __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__const__));
extern intmax_t strtoimax (const char *__restrict __nptr,
      char **__restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t strtoumax (const char *__restrict __nptr,
       char ** __restrict __endptr, int __base) __attribute__ ((__nothrow__ , __leaf__));
extern intmax_t wcstoimax (const __gwchar_t *__restrict __nptr,
      __gwchar_t **__restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
extern uintmax_t wcstoumax (const __gwchar_t *__restrict __nptr,
       __gwchar_t ** __restrict __endptr, int __base)
     __attribute__ ((__nothrow__ , __leaf__));
# 432 "/usr/include/inttypes.h" 3 4
# 95 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h" 2
# 103 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
  
# 103 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
 typedef int32_t idx_t;
# 134 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
  typedef float real_t;
# 199 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
int METIS_PartGraphRecursive(idx_t *nvtxs, idx_t *ncon, idx_t *xadj,
                  idx_t *adjncy, idx_t *vwgt, idx_t *vsize, idx_t *adjwgt,
                  idx_t *nparts, real_t *tpwgts, real_t *ubvec, idx_t *options,
                  idx_t *edgecut, idx_t *part);
int METIS_PartGraphKway(idx_t *nvtxs, idx_t *ncon, idx_t *xadj,
                  idx_t *adjncy, idx_t *vwgt, idx_t *vsize, idx_t *adjwgt,
                  idx_t *nparts, real_t *tpwgts, real_t *ubvec, idx_t *options,
                  idx_t *edgecut, idx_t *part);
int METIS_MeshToDual(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *ncommon, idx_t *numflag, idx_t **r_xadj, idx_t **r_adjncy);
int METIS_MeshToNodal(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *numflag, idx_t **r_xadj, idx_t **r_adjncy);
int METIS_PartMeshNodal(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *vwgt, idx_t *vsize, idx_t *nparts, real_t *tpwgts,
                  idx_t *options, idx_t *objval, idx_t *epart, idx_t *npart);
int METIS_PartMeshDual(idx_t *ne, idx_t *nn, idx_t *eptr, idx_t *eind,
                  idx_t *vwgt, idx_t *vsize, idx_t *ncommon, idx_t *nparts,
                  real_t *tpwgts, idx_t *options, idx_t *objval, idx_t *epart,
                  idx_t *npart);
int METIS_NodeND(idx_t *nvtxs, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
                  idx_t *options, idx_t *perm, idx_t *iperm);
int METIS_Free(void *ptr);
int METIS_SetDefaultOptions(idx_t *options);
int METIS_NodeNDP(idx_t nvtxs, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
                   idx_t npes, idx_t *options, idx_t *perm, idx_t *iperm,
                   idx_t *sizes);
int METIS_ComputeVertexSeparator(idx_t *nvtxs, idx_t *xadj, idx_t *adjncy,
                   idx_t *vwgt, idx_t *options, idx_t *sepsize, idx_t *part);
int METIS_NodeRefine(idx_t nvtxs, idx_t *xadj, idx_t *vwgt, idx_t *adjncy,
                   idx_t *where, idx_t *hmarker, real_t ubfactor);
# 255 "/home/ryan/projects/moose/petsc/linux-c-opt/include/metis.h"
typedef enum {
  METIS_OK = 1,
  METIS_ERROR_INPUT = -2,
  METIS_ERROR_MEMORY = -3,
  METIS_ERROR = -4
} rstatus_et;
typedef enum {
  METIS_OP_PMETIS,
  METIS_OP_KMETIS,
  METIS_OP_OMETIS
} moptype_et;
typedef enum {
  METIS_OPTION_PTYPE,
  METIS_OPTION_OBJTYPE,
  METIS_OPTION_CTYPE,
  METIS_OPTION_IPTYPE,
  METIS_OPTION_RTYPE,
  METIS_OPTION_DBGLVL,
  METIS_OPTION_NITER,
  METIS_OPTION_NCUTS,
  METIS_OPTION_SEED,
  METIS_OPTION_NO2HOP,
  METIS_OPTION_MINCONN,
  METIS_OPTION_CONTIG,
  METIS_OPTION_COMPRESS,
  METIS_OPTION_CCORDER,
  METIS_OPTION_PFACTOR,
  METIS_OPTION_NSEPS,
  METIS_OPTION_UFACTOR,
  METIS_OPTION_NUMBERING,
  METIS_OPTION_HELP,
  METIS_OPTION_TPWGTS,
  METIS_OPTION_NCOMMON,
  METIS_OPTION_NOOUTPUT,
  METIS_OPTION_BALANCE,
  METIS_OPTION_GTYPE,
  METIS_OPTION_UBVEC
} moptions_et;
typedef enum {
  METIS_PTYPE_RB,
  METIS_PTYPE_KWAY
} mptype_et;
typedef enum {
  METIS_GTYPE_DUAL,
  METIS_GTYPE_NODAL
} mgtype_et;
typedef enum {
  METIS_CTYPE_RM,
  METIS_CTYPE_SHEM
} mctype_et;
typedef enum {
  METIS_IPTYPE_GROW,
  METIS_IPTYPE_RANDOM,
  METIS_IPTYPE_EDGE,
  METIS_IPTYPE_NODE,
  METIS_IPTYPE_METISRB
} miptype_et;
typedef enum {
  METIS_RTYPE_FM,
  METIS_RTYPE_GREEDY,
  METIS_RTYPE_SEP2SIDED,
  METIS_RTYPE_SEP1SIDED
} mrtype_et;
typedef enum {
  METIS_DBG_INFO = 1,
  METIS_DBG_TIME = 2,
  METIS_DBG_COARSEN = 4,
  METIS_DBG_REFINE = 8,
  METIS_DBG_IPART = 16,
  METIS_DBG_MOVEINFO = 32,
  METIS_DBG_SEPINFO = 64,
  METIS_DBG_CONNINFO = 128,
  METIS_DBG_CONTIGINFO = 256,
  METIS_DBG_MEMORY = 2048,
} mdbglvl_et;
typedef enum {
  METIS_OBJTYPE_CUT,
  METIS_OBJTYPE_VOL,
  METIS_OBJTYPE_NODE
} mobjtype_et;
# 19 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h" 2
# 67 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h"
int ParMETIS_V3_PartKway(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
      idx_t *adjwgt, idx_t *wgtflag, idx_t *numflag, idx_t *ncon, idx_t *nparts,
      real_t *tpwgts, real_t *ubvec, idx_t *options, idx_t *edgecut, idx_t *part,
      MPI_Comm *comm);
int ParMETIS_V3_PartGeomKway(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
      idx_t *adjwgt, idx_t *wgtflag, idx_t *numflag, idx_t *ndims, real_t *xyz,
      idx_t *ncon, idx_t *nparts, real_t *tpwgts, real_t *ubvec, idx_t *options,
      idx_t *edgecut, idx_t *part, MPI_Comm *comm);
int ParMETIS_V3_PartGeom(
             idx_t *vtxdist, idx_t *ndims, real_t *xyz, idx_t *part, MPI_Comm *comm);
int ParMETIS_V3_RefineKway(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
      idx_t *adjwgt, idx_t *wgtflag, idx_t *numflag, idx_t *ncon, idx_t *nparts,
      real_t *tpwgts, real_t *ubvec, idx_t *options, idx_t *edgecut,
      idx_t *part, MPI_Comm *comm);
int ParMETIS_V3_AdaptiveRepart(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
      idx_t *vsize, idx_t *adjwgt, idx_t *wgtflag, idx_t *numflag, idx_t *ncon,
      idx_t *nparts, real_t *tpwgts, real_t *ubvec, real_t *ipc2redist,
      idx_t *options, idx_t *edgecut, idx_t *part, MPI_Comm *comm);
int ParMETIS_V3_Mesh2Dual(
             idx_t *elmdist, idx_t *eptr, idx_t *eind, idx_t *numflag,
      idx_t *ncommonnodes, idx_t **xadj, idx_t **adjncy, MPI_Comm *comm);
int ParMETIS_V3_PartMeshKway(
             idx_t *elmdist, idx_t *eptr, idx_t *eind, idx_t *elmwgt,
      idx_t *wgtflag, idx_t *numflag, idx_t *ncon, idx_t *ncommonnodes, idx_t *nparts,
      real_t *tpwgts, real_t *ubvec, idx_t *options, idx_t *edgecut, idx_t *part,
      MPI_Comm *comm);
int ParMETIS_V3_NodeND(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *numflag,
      idx_t *options, idx_t *order, idx_t *sizes, MPI_Comm *comm);
int ParMETIS_V32_NodeND(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *vwgt,
             idx_t *numflag, idx_t *mtype, idx_t *rtype, idx_t *p_nseps, idx_t *s_nseps,
             real_t *ubfrac, idx_t *seed, idx_t *dbglvl, idx_t *order,
             idx_t *sizes, MPI_Comm *comm);
int ParMETIS_SerialNodeND(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy, idx_t *numflag,
             idx_t *options, idx_t *order, idx_t *sizes, MPI_Comm *comm);
void ParMETIS_ComputeVertexSeparator(
             idx_t *vtxdist, idx_t *xadj, idx_t *adjncy,
             idx_t *p_nseps, idx_t *s_nseps,
             real_t *ubfrac, idx_t *idbglvl, idx_t *order, idx_t *sizes,
             MPI_Comm *comm);
# 133 "/home/ryan/projects/moose/petsc/linux-c-opt/include/parmetis.h"
typedef enum {
  PARMETIS_OP_KMETIS,
  PARMETIS_OP_GKMETIS,
  PARMETIS_OP_GMETIS,
  PARMETIS_OP_RMETIS,
  PARMETIS_OP_AMETIS,
  PARMETIS_OP_OMETIS,
  PARMETIS_OP_M2DUAL,
  PARMETIS_OP_MKMETIS
} pmoptype_et;
# 3 "/tmp/petsc-VuReVk/config.headers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
Found header files ['parmetis.h'] in ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
                  Popping language C
              Checking for functions [ParMETIS_ComputeVertexSeparator] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libparmetis.a'] ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libmetis.a', 'libm.a']
                Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ParMETIS_ComputeVertexSeparator();
static void _check_ParMETIS_ComputeVertexSeparator() { ParMETIS_ComputeVertexSeparator(); }

int main() {
_check_ParMETIS_ComputeVertexSeparator();;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lparmetis -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lmetis -lm -lstdc++ -ldl 
                Defined "HAVE_LIBPARMETIS" to "1"
                Popping language C
================================================================================
TEST checkSharedLibrary from config.packages.parmetis(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.parmetis(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.Zoltan(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Zoltan(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.Chaco(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.Chaco(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.hdf5(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.hdf5(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.med(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.med(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.cgns(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.cgns(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.ascem-io(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.ascem-io(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.pflotran(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.pflotran(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.alquimia(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.alquimia(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.SAMRAI(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.SAMRAI(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.ADIOS2(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.ADIOS2(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.pnetcdf(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.pnetcdf(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.netcdf(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.netcdf(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.exodusii(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.exodusii(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.ADIOS(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.ADIOS(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
================================================================================
TEST alternateConfigureLibrary from config.packages.openblas(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.openblas(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.fblaslapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.fblaslapack(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.fblaslapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:747)
TESTING: configureLibrary from config.packages.fblaslapack(config/BuildSystem/config/package.py:747)
  Find an installation and check if it can work with PETSc
==================================================================================
            Checking for a functional fblaslapack
                    Looking for FBLASLAPACK at git.fblaslapack, hg.fblaslapack or a directory starting with ['fblaslapack']
                    Found a copy of FBLASLAPACK in fblaslapack-3.4.2
                      Pushing language FC
Executing: mpif90 -V
                      Popping language FC
                      Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  -O0   /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main

      end
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.libraries -I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  -mfp16-format=ieee  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90 
Possible ERROR while running compiler: exit code 256
stderr:
gfortran: error: unrecognized command line option ???-mfp16-format=ieee???
Source:
      program main

      end
                        Rejecting compiler flag -mfp16-format=ieee due to nonzero status from link
                        Rejecting compiler flag -mfp16-format=ieee due to 
gfortran: error: unrecognized command line option ???-mfp16-format=ieee???

PETSc Error: No output file produced
                      Popping language FC
                      Pushing language FC
                      Popping language FC
Do not need to rebuild FBLASLAPACK
            Not checking for library in Download FBLASLAPACK: [] because no functions given to check for
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
================================================================================
TEST check from config.libraries(/home/ryan/projects/moose/petsc/config/BuildSystem/config/libraries.py:158)
TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:158)
  Checks that the library "libName" contains "funcs", and if it does defines HAVE_LIB"libName"
       - libDir may be a list of directories
       - libName may be a list of library names
                No functions to check for in library [] []
                No functions to check for in library [] []
            Checking for headers Download FBLASLAPACK: ['/home/ryan/projects/moose/petsc/linux-c-opt/include', '/usr/include', '/usr/lib/openmpi']
================================================================================
TEST checkSharedLibrary from config.packages.fblaslapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:801)
TESTING: checkSharedLibrary from config.packages.fblaslapack(config/BuildSystem/config/package.py:801)
  By default we don't care about checking if the library is shared
          Popping language C
================================================================================
TEST alternateConfigureLibrary from config.packages.f2cblaslapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:805)
TESTING: alternateConfigureLibrary from config.packages.f2cblaslapack(config/BuildSystem/config/package.py:805)
  Called if --with-packagename=0; does nothing by default
          Pushing language C
================================================================================
TEST checkDependencies from config.packages.BlasLapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py:722)
TESTING: checkDependencies from config.packages.BlasLapack(config/BuildSystem/config/package.py:722)
================================================================================
TEST configureLibrary from config.packages.BlasLapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/BlasLapack.py:356)
TESTING: configureLibrary from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:356)
================================================================================
Checking for a functional BLAS and LAPACK in fblaslapack
================================================================================
TEST checkLib from config.packages.BlasLapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/BlasLapack.py:114)
TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114)
  Checking for BLAS and LAPACK symbols
                    Checking for functions [ddot_] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libfblas.a'] ['libm.a', '-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
                      Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ddot_();
static void _check_ddot_() { ddot_(); }

int main() {
_check_ddot_();;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lfblas -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfblas
collect2: error: ld returned 1 exit status
                      Popping language C
                Checking for no name mangling on BLAS/LAPACK
                    Checking for functions [ddot] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libfblas.a'] ['libm.a', '-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
                      Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ddot();
static void _check_ddot() { ddot(); }

int main() {
_check_ddot();;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lfblas -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfblas
collect2: error: ld returned 1 exit status
                      Popping language C
                Checking for underscore name mangling on BLAS/LAPACK
                    Checking for functions [ddot_] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libfblas.a'] ['libm.a', '-lstdc++', '-ldl', '-Wl,-rpath,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread']
                      Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/tmp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ddot_();
static void _check_ddot_() { ddot_(); }

int main() {
_check_ddot_();;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lfblas -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl 
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfblas
collect2: error: ld returned 1 exit status
                      Popping language C
                Unknown name mangling in BLAS/LAPACK
**** Configure header /tmp/petsc-VuReVk/confdefs.h ****
#if !defined(INCLUDED_UNKNOWN)
#define INCLUDED_UNKNOWN

#ifndef HAVE_MATH_INFINITY
#define HAVE_MATH_INFINITY 1
#endif

#ifndef IS_COLORING_MAX
#define IS_COLORING_MAX 65535
#endif

#ifndef STDC_HEADERS
#define STDC_HEADERS 1
#endif

#ifndef MPIU_COLORING_VALUE
#define MPIU_COLORING_VALUE MPI_UNSIGNED_SHORT
#endif

#ifndef PETSC_HAVE_GZIP
#define PETSC_HAVE_GZIP 1
#endif

#ifndef PETSC_HAVE_STRING_H
#define PETSC_HAVE_STRING_H 1
#endif

#ifndef PETSC_HAVE_SYS_TIMES_H
#define PETSC_HAVE_SYS_TIMES_H 1
#endif

#ifndef PETSC_HAVE_SYS_TYPES_H
#define PETSC_HAVE_SYS_TYPES_H 1
#endif

#ifndef PETSC_HAVE_ENDIAN_H
#define PETSC_HAVE_ENDIAN_H 1
#endif

#ifndef PETSC_HAVE_SYS_PROCFS_H
#define PETSC_HAVE_SYS_PROCFS_H 1
#endif

#ifndef PETSC_HAVE_DLFCN_H
#define PETSC_HAVE_DLFCN_H 1
#endif

#ifndef PETSC_HAVE_SCHED_H
#define PETSC_HAVE_SCHED_H 1
#endif

#ifndef PETSC_HAVE_STDINT_H
#define PETSC_HAVE_STDINT_H 1
#endif

#ifndef PETSC_HAVE_LINUX_KERNEL_H
#define PETSC_HAVE_LINUX_KERNEL_H 1
#endif

#ifndef PETSC_HAVE_TIME_H
#define PETSC_HAVE_TIME_H 1
#endif

#ifndef PETSC_HAVE_MATH_H
#define PETSC_HAVE_MATH_H 1
#endif

#ifndef PETSC_HAVE_INTTYPES_H
#define PETSC_HAVE_INTTYPES_H 1
#endif

#ifndef PETSC_HAVE_STDLIB_H
#define PETSC_HAVE_STDLIB_H 1
#endif

#ifndef PETSC_HAVE_SYS_PARAM_H
#define PETSC_HAVE_SYS_PARAM_H 1
#endif

#ifndef PETSC_HAVE_PTHREAD_H
#define PETSC_HAVE_PTHREAD_H 1
#endif

#ifndef PETSC_HAVE_UNISTD_H
#define PETSC_HAVE_UNISTD_H 1
#endif

#ifndef PETSC_HAVE_SYS_WAIT_H
#define PETSC_HAVE_SYS_WAIT_H 1
#endif

#ifndef PETSC_HAVE_SETJMP_H
#define PETSC_HAVE_SETJMP_H 1
#endif

#ifndef PETSC_HAVE_LIMITS_H
#define PETSC_HAVE_LIMITS_H 1
#endif

#ifndef PETSC_HAVE_SYS_UTSNAME_H
#define PETSC_HAVE_SYS_UTSNAME_H 1
#endif

#ifndef PETSC_HAVE_NETINET_IN_H
#define PETSC_HAVE_NETINET_IN_H 1
#endif

#ifndef PETSC_HAVE_SYS_SOCKET_H
#define PETSC_HAVE_SYS_SOCKET_H 1
#endif

#ifndef PETSC_HAVE_FLOAT_H
#define PETSC_HAVE_FLOAT_H 1
#endif

#ifndef PETSC_HAVE_SEARCH_H
#define PETSC_HAVE_SEARCH_H 1
#endif

#ifndef PETSC_HAVE_SYS_RESOURCE_H
#define PETSC_HAVE_SYS_RESOURCE_H 1
#endif

#ifndef PETSC_TIME_WITH_SYS_TIME
#define PETSC_TIME_WITH_SYS_TIME 1
#endif

#ifndef PETSC_HAVE_NETDB_H
#define PETSC_HAVE_NETDB_H 1
#endif

#ifndef PETSC_HAVE_MALLOC_H
#define PETSC_HAVE_MALLOC_H 1
#endif

#ifndef PETSC_HAVE_PWD_H
#define PETSC_HAVE_PWD_H 1
#endif

#ifndef PETSC_HAVE_FCNTL_H
#define PETSC_HAVE_FCNTL_H 1
#endif

#ifndef PETSC_HAVE_STRINGS_H
#define PETSC_HAVE_STRINGS_H 1
#endif

#ifndef PETSC_HAVE_IMMINTRIN_H
#define PETSC_HAVE_IMMINTRIN_H 1
#endif

#ifndef PETSC_HAVE_SYS_SYSINFO_H
#define PETSC_HAVE_SYS_SYSINFO_H 1
#endif

#ifndef PETSC_HAVE_SYS_TIME_H
#define PETSC_HAVE_SYS_TIME_H 1
#endif

#ifndef PETSC_HAVE_C99
#define PETSC_HAVE_C99 1
#endif

#ifndef PETSC_USING_F90
#define PETSC_USING_F90 1
#endif

#ifndef PETSC_USING_F2003
#define PETSC_USING_F2003 1
#endif

#ifndef PETSC_FORTRAN_CHARLEN_T
#define PETSC_FORTRAN_CHARLEN_T size_t
#endif

#ifndef PETSC_HAVE_RTLD_NOW
#define PETSC_HAVE_RTLD_NOW 1
#endif

#ifndef PETSC_HAVE_RTLD_LOCAL
#define PETSC_HAVE_RTLD_LOCAL 1
#endif

#ifndef PETSC_HAVE_FORTRAN_TYPE_INITIALIZE
#define PETSC_HAVE_FORTRAN_TYPE_INITIALIZE -2
#endif

#ifndef PETSC_HAVE_RTLD_LAZY
#define PETSC_HAVE_RTLD_LAZY 1
#endif

#ifndef PETSC_CXX_INLINE
#define PETSC_CXX_INLINE inline
#endif

#ifndef PETSC_HAVE_FORTRAN_FLUSH
#define PETSC_HAVE_FORTRAN_FLUSH 1
#endif

#ifndef PETSC_HAVE_FORTRAN_UNDERSCORE
#define PETSC_HAVE_FORTRAN_UNDERSCORE 1
#endif

#ifndef PETSC_HAVE_CXX_NAMESPACE
#define PETSC_HAVE_CXX_NAMESPACE 1
#endif

#ifndef PETSC_HAVE_RTLD_GLOBAL
#define PETSC_HAVE_RTLD_GLOBAL 1
#endif

#ifndef PETSC_C_RESTRICT
#define PETSC_C_RESTRICT __restrict
#endif

#ifndef PETSC_FORTRAN_TYPE_INITIALIZE
#define PETSC_FORTRAN_TYPE_INITIALIZE  = -2
#endif

#ifndef PETSC_HAVE_FORTRAN_TYPE_STAR
#define PETSC_HAVE_FORTRAN_TYPE_STAR 1
#endif

#ifndef PETSC_CXX_RESTRICT
#define PETSC_CXX_RESTRICT __restrict
#endif

#ifndef PETSC_C_INLINE
#define PETSC_C_INLINE inline
#endif

#ifndef PETSC_HAVE_LIBDL
#define PETSC_HAVE_LIBDL 1
#endif

#ifndef PETSC_HAVE_LIBM
#define PETSC_HAVE_LIBM 1
#endif

#ifndef PETSC_HAVE_LIBMETIS
#define PETSC_HAVE_LIBMETIS 1
#endif

#ifndef PETSC_HAVE_FENV_H
#define PETSC_HAVE_FENV_H 1
#endif

#ifndef PETSC_HAVE_LOG2
#define PETSC_HAVE_LOG2 1
#endif

#ifndef PETSC_HAVE_LIBPARMETIS
#define PETSC_HAVE_LIBPARMETIS 1
#endif

#ifndef PETSC_HAVE_ERF
#define PETSC_HAVE_ERF 1
#endif

#ifndef PETSC_HAVE_LIBX11
#define PETSC_HAVE_LIBX11 1
#endif

#ifndef PETSC_HAVE_TGAMMA
#define PETSC_HAVE_TGAMMA 1
#endif

#ifndef PETSC_USE_SCALAR_REAL
#define PETSC_USE_SCALAR_REAL 1
#endif

#ifndef PETSC_HAVE_ISINF
#define PETSC_HAVE_ISINF 1
#endif

#ifndef PETSC_HAVE_ISNAN
#define PETSC_HAVE_ISNAN 1
#endif

#ifndef PETSC_HAVE_ISNORMAL
#define PETSC_HAVE_ISNORMAL 1
#endif

#ifndef PETSC_USE_REAL_DOUBLE
#define PETSC_USE_REAL_DOUBLE 1
#endif

#ifndef PETSC_SIZEOF_MPI_COMM
#define PETSC_SIZEOF_MPI_COMM 8
#endif

#ifndef PETSC_BITS_PER_BYTE
#define PETSC_BITS_PER_BYTE 8
#endif

#ifndef PETSC_SIZEOF_MPI_FINT
#define PETSC_SIZEOF_MPI_FINT 4
#endif

#ifndef PETSC_USE_VISIBILITY_C
#define PETSC_USE_VISIBILITY_C 1
#endif

#ifndef PETSC_SIZEOF_VOID_P
#define PETSC_SIZEOF_VOID_P 8
#endif

#ifndef PETSC_RETSIGTYPE
#define PETSC_RETSIGTYPE void
#endif

#ifndef PETSC_HAVE_CXX_COMPLEX
#define PETSC_HAVE_CXX_COMPLEX 1
#endif

#ifndef PETSC_SIZEOF_LONG
#define PETSC_SIZEOF_LONG 8
#endif

#ifndef PETSC_USE_FORTRANKIND
#define PETSC_USE_FORTRANKIND 1
#endif

#ifndef PETSC_USE_VISIBILITY_CXX
#define PETSC_USE_VISIBILITY_CXX 1
#endif

#ifndef PETSC_SIZEOF_SIZE_T
#define PETSC_SIZEOF_SIZE_T 8
#endif

#ifndef PETSC_HAVE_SIGINFO_T
#define PETSC_HAVE_SIGINFO_T 1
#endif

#ifndef PETSC_SIZEOF_CHAR
#define PETSC_SIZEOF_CHAR 1
#endif

#ifndef PETSC_SIZEOF_DOUBLE
#define PETSC_SIZEOF_DOUBLE 8
#endif

#ifndef PETSC_SIZEOF_FLOAT
#define PETSC_SIZEOF_FLOAT 4
#endif

#ifndef PETSC_HAVE_C99_COMPLEX
#define PETSC_HAVE_C99_COMPLEX 1
#endif

#ifndef PETSC_SIZEOF_INT
#define PETSC_SIZEOF_INT 4
#endif

#ifndef PETSC_SIZEOF_LONG_LONG
#define PETSC_SIZEOF_LONG_LONG 8
#endif

#ifndef PETSC_SIZEOF_SHORT
#define PETSC_SIZEOF_SHORT 2
#endif

#ifndef PETSC_CLANGUAGE_C
#define PETSC_CLANGUAGE_C 1
#endif

#ifndef PETSC_HAVE_STRCASECMP
#define PETSC_HAVE_STRCASECMP 1
#endif

#ifndef PETSC_HAVE_GET_NPROCS
#define PETSC_HAVE_GET_NPROCS 1
#endif

#ifndef PETSC_HAVE_POPEN
#define PETSC_HAVE_POPEN 1
#endif

#ifndef PETSC_HAVE_SIGSET
#define PETSC_HAVE_SIGSET 1
#endif

#ifndef PETSC_HAVE_GETWD
#define PETSC_HAVE_GETWD 1
#endif

#ifndef PETSC_HAVE_VSNPRINTF
#define PETSC_HAVE_VSNPRINTF 1
#endif

#ifndef PETSC_HAVE_TIMES
#define PETSC_HAVE_TIMES 1
#endif

#ifndef PETSC_HAVE_DLSYM
#define PETSC_HAVE_DLSYM 1
#endif

#ifndef PETSC_HAVE_SNPRINTF
#define PETSC_HAVE_SNPRINTF 1
#endif

#ifndef PETSC_HAVE_GETHOSTBYNAME
#define PETSC_HAVE_GETHOSTBYNAME 1
#endif

#ifndef PETSC_HAVE_GETCWD
#define PETSC_HAVE_GETCWD 1
#endif

#ifndef PETSC_HAVE_DLERROR
#define PETSC_HAVE_DLERROR 1
#endif

#ifndef PETSC_HAVE_FORK
#define PETSC_HAVE_FORK 1
#endif

#ifndef PETSC_HAVE_RAND
#define PETSC_HAVE_RAND 1
#endif

#ifndef PETSC_HAVE_GETTIMEOFDAY
#define PETSC_HAVE_GETTIMEOFDAY 1
#endif

#ifndef PETSC_HAVE_DLCLOSE
#define PETSC_HAVE_DLCLOSE 1
#endif

#ifndef PETSC_HAVE_UNAME
#define PETSC_HAVE_UNAME 1
#endif

#ifndef PETSC_HAVE_GETHOSTNAME
#define PETSC_HAVE_GETHOSTNAME 1
#endif

#ifndef PETSC_HAVE_MKSTEMP
#define PETSC_HAVE_MKSTEMP 1
#endif

#ifndef PETSC_HAVE_SIGACTION
#define PETSC_HAVE_SIGACTION 1
#endif

#ifndef PETSC_HAVE_DRAND48
#define PETSC_HAVE_DRAND48 1
#endif

#ifndef PETSC_HAVE_MEMALIGN
#define PETSC_HAVE_MEMALIGN 1
#endif

#ifndef PETSC_HAVE_VA_COPY
#define PETSC_HAVE_VA_COPY 1
#endif

#ifndef PETSC_HAVE_CLOCK
#define PETSC_HAVE_CLOCK 1
#endif

#ifndef PETSC_HAVE_ACCESS
#define PETSC_HAVE_ACCESS 1
#endif

#ifndef PETSC_HAVE_SIGNAL
#define PETSC_HAVE_SIGNAL 1
#endif

#ifndef PETSC_HAVE_USLEEP
#define PETSC_HAVE_USLEEP 1
#endif

#ifndef PETSC_HAVE_GETRUSAGE
#define PETSC_HAVE_GETRUSAGE 1
#endif

#ifndef PETSC_HAVE_VFPRINTF
#define PETSC_HAVE_VFPRINTF 1
#endif

#ifndef PETSC_HAVE_NANOSLEEP
#define PETSC_HAVE_NANOSLEEP 1
#endif

#ifndef PETSC_HAVE_GETDOMAINNAME
#define PETSC_HAVE_GETDOMAINNAME 1
#endif

#ifndef PETSC_HAVE_TIME
#define PETSC_HAVE_TIME 1
#endif

#ifndef PETSC_HAVE_LSEEK
#define PETSC_HAVE_LSEEK 1
#endif

#ifndef PETSC_HAVE_SOCKET
#define PETSC_HAVE_SOCKET 1
#endif

#ifndef PETSC_HAVE_SYSINFO
#define PETSC_HAVE_SYSINFO 1
#endif

#ifndef PETSC_HAVE_READLINK
#define PETSC_HAVE_READLINK 1
#endif

#ifndef PETSC_HAVE_REALPATH
#define PETSC_HAVE_REALPATH 1
#endif

#ifndef PETSC_HAVE_DLOPEN
#define PETSC_HAVE_DLOPEN 1
#endif

#ifndef PETSC_HAVE_MEMMOVE
#define PETSC_HAVE_MEMMOVE 1
#endif

#ifndef PETSC_HAVE__GFORTRAN_IARGC
#define PETSC_HAVE__GFORTRAN_IARGC 1
#endif

#ifndef PETSC_HAVE_MMAP
#define PETSC_HAVE_MMAP 1
#endif

#ifndef PETSC_HAVE_GETPAGESIZE
#define PETSC_HAVE_GETPAGESIZE 1
#endif

#ifndef PETSC_HAVE_SLEEP
#define PETSC_HAVE_SLEEP 1
#endif

#ifndef PETSC_HAVE_VPRINTF
#define PETSC_HAVE_VPRINTF 1
#endif

#ifndef PETSC_HAVE_BZERO
#define PETSC_HAVE_BZERO 1
#endif

#ifndef PETSC_SIGNAL_CAST
#define PETSC_SIGNAL_CAST  
#endif

#ifndef PETSC_WRITE_MEMORY_BARRIER
#define PETSC_WRITE_MEMORY_BARRIER() __asm__ __volatile__ ("sfence":::"memory")
#endif

#ifndef PETSC_MEMORY_BARRIER
#define PETSC_MEMORY_BARRIER() __asm__ __volatile__ ("mfence":::"memory")
#endif

#ifndef PETSC_READ_MEMORY_BARRIER
#define PETSC_READ_MEMORY_BARRIER() __asm__ __volatile__ ("lfence":::"memory")
#endif

#ifndef PETSC_CPU_RELAX
#define PETSC_CPU_RELAX() __asm__ __volatile__("rep; nop" ::: "memory")
#endif

#ifndef PETSC_HAVE_DYNAMIC_LIBRARIES
#define PETSC_HAVE_DYNAMIC_LIBRARIES 1
#endif

#ifndef PETSC_HAVE_SHARED_LIBRARIES
#define PETSC_HAVE_SHARED_LIBRARIES 1
#endif

#ifndef PETSC_USE_SHARED_LIBRARIES
#define PETSC_USE_SHARED_LIBRARIES 1
#endif

#ifndef PETSC_USE_GDB_DEBUGGER
#define PETSC_USE_GDB_DEBUGGER 1
#endif

#ifndef PETSC_VERSION_DATE_GIT
#define PETSC_VERSION_DATE_GIT "2018-12-02 07:19:50 +0100"
#endif

#ifndef PETSC_VERSION_BRANCH_GIT
#define PETSC_VERSION_BRANCH_GIT "(HEAD detached at e1487ad806)"
#endif

#ifndef PETSC_VERSION_GIT
#define PETSC_VERSION_GIT "v3.10.2-76-ge1487ad806"
#endif

#ifndef PETSC_HAVE_MPI_F90MODULE
#define PETSC_HAVE_MPI_F90MODULE 1
#endif

#ifndef PETSC_HAVE_MPI_WIN_CREATE_FEATURE
#define PETSC_HAVE_MPI_WIN_CREATE_FEATURE 1
#endif

#ifndef PETSC_HAVE_MPI_IALLREDUCE
#define PETSC_HAVE_MPI_IALLREDUCE 1
#endif

#ifndef PETSC_HAVE_OMPI_MINOR_VERSION
#define PETSC_HAVE_OMPI_MINOR_VERSION 0
#endif

#ifndef PETSC_HAVE_MPI_REDUCE_SCATTER_BLOCK
#define PETSC_HAVE_MPI_REDUCE_SCATTER_BLOCK 1
#endif

#ifndef PETSC_HAVE_MPI_IN_PLACE
#define PETSC_HAVE_MPI_IN_PLACE 1
#endif

#ifndef PETSC_HAVE_MPI_COMM_C2F
#define PETSC_HAVE_MPI_COMM_C2F 1
#endif

#ifndef PETSC_HAVE_MPI_GET_LIBRARY_VERSION
#define PETSC_HAVE_MPI_GET_LIBRARY_VERSION 1
#endif

#ifndef PETSC_HAVE_MPI_SHARED_COMM
#define PETSC_HAVE_MPI_SHARED_COMM 1
#endif

#ifndef PETSC_HAVE_MPI_TYPE_GET_EXTENT
#define PETSC_HAVE_MPI_TYPE_GET_EXTENT 1
#endif

#ifndef PETSC_HAVE_MPI_WIN_CREATE
#define PETSC_HAVE_MPI_WIN_CREATE 1
#endif

#ifndef PETSC_HAVE_MPIIO
#define PETSC_HAVE_MPIIO 1
#endif

#ifndef PETSC_HAVE_MPI_INIT_THREAD
#define PETSC_HAVE_MPI_INIT_THREAD 1
#endif

#ifndef PETSC_HAVE_MPI_COMBINER_NAMED
#define PETSC_HAVE_MPI_COMBINER_NAMED 1
#endif

#ifndef PETSC_HAVE_MPI_COMM_F2C
#define PETSC_HAVE_MPI_COMM_F2C 1
#endif

#ifndef PETSC_HAVE_MPI_PROCESS_SHARED_MEMORY
#define PETSC_HAVE_MPI_PROCESS_SHARED_MEMORY 1
#endif

#ifndef PETSC_HAVE_MPI_TYPE_GET_ENVELOPE
#define PETSC_HAVE_MPI_TYPE_GET_ENVELOPE 1
#endif

#ifndef PETSC_HAVE_OMPI_MAJOR_VERSION
#define PETSC_HAVE_OMPI_MAJOR_VERSION 4
#endif

#ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
#define PETSC_HAVE_MPI_REDUCE_SCATTER 1
#endif

#ifndef PETSC_HAVE_MPI_COMBINER_DUP
#define PETSC_HAVE_MPI_COMBINER_DUP 1
#endif

#ifndef PETSC_HAVE_MPI_TYPE_DUP
#define PETSC_HAVE_MPI_TYPE_DUP 1
#endif

#ifndef PETSC_HAVE_MPI_COMM_SPAWN
#define PETSC_HAVE_MPI_COMM_SPAWN 1
#endif

#ifndef PETSC_HAVE_MPI_FINT
#define PETSC_HAVE_MPI_FINT 1
#endif

#ifndef PETSC_HAVE_MPI_IBARRIER
#define PETSC_HAVE_MPI_IBARRIER 1
#endif

#ifndef PETSC_HAVE_MPI_FINALIZED
#define PETSC_HAVE_MPI_FINALIZED 1
#endif

#ifndef PETSC_HAVE_MPI_COMBINER_CONTIGUOUS
#define PETSC_HAVE_MPI_COMBINER_CONTIGUOUS 1
#endif

#ifndef PETSC_HAVE_MPI_WIN_SHARED_QUERY
#define PETSC_HAVE_MPI_WIN_SHARED_QUERY 1
#endif

#ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
#define PETSC_HAVE_OMPI_RELEASE_VERSION 0
#endif

#ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
#define PETSC_HAVE_MPI_REDUCE_LOCAL 1
#endif

#ifndef PETSC_HAVE_MPI_REPLACE
#define PETSC_HAVE_MPI_REPLACE 1
#endif

#ifndef PETSC_HAVE_MPI_EXSCAN
#define PETSC_HAVE_MPI_EXSCAN 1
#endif

#ifndef PETSC_HAVE_MPI_ALLTOALLW
#define PETSC_HAVE_MPI_ALLTOALLW 1
#endif

#ifndef PETSC_HAVE_MPI_WIN_ALLOCATE_SHARED
#define PETSC_HAVE_MPI_WIN_ALLOCATE_SHARED 1
#endif

#ifndef PETSC_USE_INFO
#define PETSC_USE_INFO 1
#endif

#ifndef PETSC_Alignx
#define PETSC_Alignx(a,b)   
#endif

#ifndef PETSC_USE_BACKWARD_LOOP
#define PETSC_USE_BACKWARD_LOOP 1
#endif

#ifndef PETSC_USE_MALLOC_COALESCED
#define PETSC_USE_MALLOC_COALESCED 1
#endif

#ifndef PETSC_USE_LOG
#define PETSC_USE_LOG 1
#endif

#ifndef PETSC_IS_COLOR_VALUE_TYPE_F
#define PETSC_IS_COLOR_VALUE_TYPE_F integer2
#endif

#ifndef PETSC_IS_COLOR_VALUE_TYPE
#define PETSC_IS_COLOR_VALUE_TYPE short
#endif

#ifndef PETSC_USE_CTABLE
#define PETSC_USE_CTABLE 1
#endif

#ifndef PETSC_MEMALIGN
#define PETSC_MEMALIGN 16
#endif

#ifndef PETSC_LEVEL1_DCACHE_LINESIZE
#define PETSC_LEVEL1_DCACHE_LINESIZE 64
#endif

#ifndef PETSC_LEVEL1_DCACHE_SIZE
#define PETSC_LEVEL1_DCACHE_SIZE 32768
#endif

#ifndef PETSC_LEVEL1_DCACHE_ASSOC
#define PETSC_LEVEL1_DCACHE_ASSOC 8
#endif

#ifndef PETSC__GNU_SOURCE
#define PETSC__GNU_SOURCE 1
#endif

#ifndef PETSC__BSD_SOURCE
#define PETSC__BSD_SOURCE 1
#endif

#ifndef PETSC__DEFAULT_SOURCE
#define PETSC__DEFAULT_SOURCE 1
#endif

#ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
#define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
#endif

#ifndef PETSC_HAVE_GFORTRAN_IARGC
#define PETSC_HAVE_GFORTRAN_IARGC 1
#endif

#ifndef PETSC_USE_PROC_FOR_SIZE
#define PETSC_USE_PROC_FOR_SIZE 1
#endif

#ifndef PETSC_HAVE_SCHED_CPU_SET_T
#define PETSC_HAVE_SCHED_CPU_SET_T 1
#endif

#ifndef PETSC_HAVE_PTHREAD_BARRIER_T
#define PETSC_HAVE_PTHREAD_BARRIER_T 1
#endif

#ifndef PETSC_HAVE_SYS_SYSCTL_H
#define PETSC_HAVE_SYS_SYSCTL_H 1
#endif

#endif
**** C specific Configure header /tmp/petsc-VuReVk/conffix.h ****
#if !defined(INCLUDED_UNKNOWN)
#define INCLUDED_UNKNOWN

#if defined(__cplusplus)
extern "C" {
}
#else
#endif
#endif
*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
--download-fblaslapack libraries cannot be used
*******************************************************************************
  File "./config/configure.py", line 394, in petsc_configure
    framework.configure(out = sys.stdout)
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1092, in configure
    self.processChildren()
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1081, in processChildren
    self.serialEvaluation(self.childGraph)
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1062, in serialEvaluation
    child.configure()
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/package.py", line 866, in configure
    self.executeTest(self.configureLibrary)
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/base.py", line 127, in executeTest
    ret = test(*args,**kargs)
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/BlasLapack.py", line 364, in configureLibrary
    for (name, blasLibrary, lapackLibrary) in self.generateGuesses():
  File "/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/BlasLapack.py", line 176, in generateGuesses
    raise RuntimeError('--download-fblaslapack libraries cannot be used')
================================================================================
Finishing configure run at Sat, 23 Feb 2019 15:56:38 -0500
================================================================================

From balay at mcs.anl.gov  Wed Feb 27 12:04:00 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 27 Feb 2019 18:04:00 +0000
Subject: [petsc-users] Fwd: PETSC installation issues
In-Reply-To: <CAN5Wd-KAP+Bz59CmEavGvz-XtNq5nCErPSVR03TToCL_95FvrA@mail.gmail.com>
References: <09180386-e88b-4776-94cc-1f0cc00e00f6@googlegroups.com>
	<CAN5Wd-KAP+Bz59CmEavGvz-XtNq5nCErPSVR03TToCL_95FvrA@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1902271201540.1551@sb>

>>>
stderr:
/usr/bin/ld: cannot find -lfblas
<<<

Looks like a broken/incomplete build - that configure thinks is successful.

Suggest doing a fresh build after 'rm -rf /home/ryan/projects/moose/petsc/linux-c-opt'

Also if using threads - might want to use --download-openblas [or MKL]

Satish

On Wed, 27 Feb 2019, Fande Kong wrote:

> Hi Satish,
> 
> Do you have any idea why the configure failed?
> 
> Thanks,
> 
> Fande
> 
> On Mon, Feb 25, 2019 at 2:57 PM <ryan.c.cooper at uconn.edu> wrote:
> 
> > Greetings,
> > I am having trouble installing Petsc with the given configuration options
> > in the scripts/update_and_rebuild_petsc.sh. I have attached the
> > confiugure.log. I am running Arch linux - kernel 4.20.12, mpi: gcc 8.2.1,
> > mpif77/90: gnu fortran gcc 8.2.1.
> >
> > The main problem is that I keep getting errors for superlu_dist and
> > scalapack that the downloaded file cannot be installed with the given
> > configuration. Are there some other flags I can specify to compile and link
> > the packages with Petsc?
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "moose-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to moose-users+unsubscribe at googlegroups.com.
> > Visit this group at https://groups.google.com/group/moose-users.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com
> > <https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .
> > For more options, visit https://groups.google.com/d/optout.
> >
> 


From jczhang at mcs.anl.gov  Wed Feb 27 12:49:51 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Wed, 27 Feb 2019 18:49:51 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/86f8ff1b/attachment.html>

From sajidsyed2021 at u.northwestern.edu  Wed Feb 27 16:15:13 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 27 Feb 2019 16:15:13 -0600
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
Message-ID: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>

Hi,

I ran a TS integrator for 25 steps on a Broadwell-Xeon and Xeon-Phi (KNL).
The problem size is 5000x5000 and I'm using scalar=complex.

The program takes 125 seconds to run on Xeon and 451 seconds on KNL !

The first thing I want to change is to convert the memory access for the
program on KNL from DRAM to MCDRAM. I did run the problem in an interactive
SLURM job and specified -C quad,flat and yet I see DRAM is being used.

I'm attaching the PETSc log files and Intel APS reports as well. Any help
on how I should change my runtime parameters on KNL will be highly
appreciated. Thanks in advance.

-- 
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/b37859ea/attachment-0003.html>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/b37859ea/attachment-0004.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bdw_petsc
Type: application/octet-stream
Size: 24909 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/b37859ea/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: knl_petsc
Type: application/octet-stream
Size: 24296 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/b37859ea/attachment-0003.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/b37859ea/attachment-0005.html>

From alexlindsay239 at gmail.com  Wed Feb 27 16:20:12 2019
From: alexlindsay239 at gmail.com (Alexander Lindsay)
Date: Wed, 27 Feb 2019 14:20:12 -0800
Subject: [petsc-users] Fwd: PETSC installation issues
In-Reply-To: <alpine.LFD.2.21.1902271201540.1551@sb>
References: <09180386-e88b-4776-94cc-1f0cc00e00f6@googlegroups.com>
	<CAN5Wd-KAP+Bz59CmEavGvz-XtNq5nCErPSVR03TToCL_95FvrA@mail.gmail.com>
	<alpine.LFD.2.21.1902271201540.1551@sb>
Message-ID: <CANFcJrGj+GYz_XPKyZs=2=52zgf_8j_V0tmxqJ6jWkD1fzysnQ@mail.gmail.com>

Is this part of the log relevant?

            Checking for a functional fblaslapack
                    Looking for FBLASLAPACK at git.fblaslapack,
hg.fblaslapack or a directory starting with ['fblaslapack']
                    Found a copy of FBLASLAPACK in fblaslapack-3.4.2
                      Pushing language FC
Executing: mpif90 -V
                      Popping language FC
                      Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o
-I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types
-I/tmp/petsc-VuReVk/config.libraries
-I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O  -O0
/tmp/petsc-VuReVk/config.setCompilers/conftest.F90
Successful compile:
Source:
      program main

      end
Executing: mpif90 -c -o /tmp/petsc-VuReVk/config.setCompilers/conftest.o
-I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk/config.types
-I/tmp/petsc-VuReVk/config.libraries
-I/tmp/petsc-VuReVk/config.setCompilers -fPIC -fopenmp  -g -O
-mfp16-format=ieee  /tmp/petsc-VuReVk/config.setCompilers/conftest.F90
Possible ERROR while running compiler: exit code 256
stderr:
*gfortran: error: unrecognized command line option ?-mfp16-format=ieee?*
Source:
      program main

      end
                        Rejecting compiler flag -mfp16-format=ieee due to
nonzero status from link
                        Rejecting compiler flag -mfp16-format=ieee due to
gfortran: error: unrecognized command line option ?-mfp16-format=ieee?

PETSc Error: No output file produced
                      Popping language FC
                      Pushing language FC
                      Popping language FC
Do not need to rebuild FBLASLAPACK
            Not checking for library in Download FBLASLAPACK: [] because no
functions given to check for


On Wed, Feb 27, 2019 at 10:04 AM Balay, Satish via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> >>>
> stderr:
> /usr/bin/ld: cannot find -lfblas
> <<<
>
> Looks like a broken/incomplete build - that configure thinks is successful.
>
> Suggest doing a fresh build after 'rm -rf
> /home/ryan/projects/moose/petsc/linux-c-opt'
>
> Also if using threads - might want to use --download-openblas [or MKL]
>
> Satish
>
> On Wed, 27 Feb 2019, Fande Kong wrote:
>
> > Hi Satish,
> >
> > Do you have any idea why the configure failed?
> >
> > Thanks,
> >
> > Fande
> >
> > On Mon, Feb 25, 2019 at 2:57 PM <ryan.c.cooper at uconn.edu> wrote:
> >
> > > Greetings,
> > > I am having trouble installing Petsc with the given configuration
> options
> > > in the scripts/update_and_rebuild_petsc.sh. I have attached the
> > > confiugure.log. I am running Arch linux - kernel 4.20.12, mpi: gcc
> 8.2.1,
> > > mpif77/90: gnu fortran gcc 8.2.1.
> > >
> > > The main problem is that I keep getting errors for superlu_dist and
> > > scalapack that the downloaded file cannot be installed with the given
> > > configuration. Are there some other flags I can specify to compile and
> link
> > > the packages with Petsc?
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "moose-users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an
> > > email to moose-users+unsubscribe at googlegroups.com.
> > > Visit this group at https://groups.google.com/group/moose-users.
> > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com
> > > <
> https://groups.google.com/d/msgid/moose-users/09180386-e88b-4776-94cc-1f0cc00e00f6%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > > .
> > > For more options, visit https://groups.google.com/d/optout.
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/c5eae89f/attachment.html>

From balay at mcs.anl.gov  Wed Feb 27 16:32:17 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 27 Feb 2019 22:32:17 +0000
Subject: [petsc-users] Fwd: PETSC installation issues
In-Reply-To: <CANFcJrGj+GYz_XPKyZs=2=52zgf_8j_V0tmxqJ6jWkD1fzysnQ@mail.gmail.com>
References: <09180386-e88b-4776-94cc-1f0cc00e00f6@googlegroups.com>
	<CAN5Wd-KAP+Bz59CmEavGvz-XtNq5nCErPSVR03TToCL_95FvrA@mail.gmail.com>
	<alpine.LFD.2.21.1902271201540.1551@sb>
	<CANFcJrGj+GYz_XPKyZs=2=52zgf_8j_V0tmxqJ6jWkD1fzysnQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1902271625460.2786@sb>

On Wed, 27 Feb 2019, Alexander Lindsay wrote:

> Is this part of the log relevant?

Here is the relevant part.

>>>>>>>>
Do not need to rebuild FBLASLAPACK
<<<<<<<

Here configure thinks FBLASLAPACK is already built.

>>>>>>>>>
TEST checkLib from config.packages.BlasLapack(/home/ryan/projects/moose/petsc/config/BuildSystem/config/packages/BlasLapack.py:114)
TESTING: checkLib from config.packages.BlasLapack(config/BuildSystem/config/packages/BlasLapack.py:114)
  Checking for BLAS and LAPACK symbols
                    Checking for functions [ddot_] in library ['/home/ryan/projects/moose/petsc/linux-c-opt/lib/libfblas.a'] ['libm.a', '-lstdc++', '-ldl', '-Wl,-rpa\
th,/usr/lib/openmpi', '-L/usr/lib/openmpi', '-lmpi_usempif08', '-lmpi_usempi_ignore_tkr', '-lmpi_mpifh', '-lmpi', '-lgfortran', '-lm', '-Wl,-rpath,/usr/lib/gcc/x86_6\
4-pc-linux-gnu/8.2.1', '-L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1', '-Wl,-rpath,/usr/lib/openmpi', '-lgfortran', '-lm', '-lgomp', '-lgcc_s', '-lquadmath', '-lpthread'\
]
                      Pushing language C
Executing: mpicc -c -o /tmp/petsc-VuReVk/config.libraries/conftest.o -I/tmp/petsc-VuReVk/config.setCompilers -I/tmp/petsc-VuReVk/config.compilers -I/tmp/petsc-VuReVk\
/config.utilities.closure -I/tmp/petsc-VuReVk/config.headers -I/tmp/petsc-VuReVk/config.utilities.cacheDetails -I/tmp/petsc-VuReVk/config.atomics -I/tmp/petsc-VuReVk\
/config.functions -I/tmp/petsc-VuReVk/config.utilities.featureTestMacros -I/tmp/petsc-VuReVk/config.utilities.missing -I/tmp/petsc-VuReVk/PETSc.options.scalarTypes -\
I/tmp/petsc-VuReVk/config.packages.MPI -I/tmp/petsc-VuReVk/config.types -I/tmp/petsc-VuReVk/config.packages.valgrind -I/tmp/petsc-VuReVk/config.packages.pthread -I/t\
mp/petsc-VuReVk/config.packages.metis -I/tmp/petsc-VuReVk/config.libraries -fPIC -fopenmp   -g -O  /tmp/petsc-VuReVk/config.libraries/conftest.c
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
char ddot_();
static void _check_ddot_() { ddot_(); }

int main() {
_check_ddot_();;
  return 0;
}
                              Pushing language C
                              Popping language C
Executing: mpicc  -o /tmp/petsc-VuReVk/config.libraries/conftest   -fPIC -fopenmp   -g -O /tmp/petsc-VuReVk/config.libraries/conftest.o  -Wl,-rpath,/home/ryan/projects/moose/petsc/linux-c-opt/lib -L/home/ryan/projects/moose/petsc/linux-c-opt/lib -lfblas -lm -lstdc++ -ldl -Wl,-rpath,/usr/lib/openmpi -L/usr/lib/openmpi -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1 -Wl,-rpath,/usr/lib/openmpi -lgfortran -lm -lgomp -lgcc_s -lquadmath -lpthread -lstdc++ -ldl
Possible ERROR while running linker: exit code 256
stderr:
/usr/bin/ld: cannot find -lfblas
collect2: error: ld returned 1 exit status
<<<<<<<<<<

But here - the compiler is not finding the library.

So the library is not likely built.

You can either do:

rm -rf /home/ryan/projects/moose/petsc/linux-c-opt/

i.e delete all build files - and start a fresh build.

Or delete the file configure looks for and assumes this package is built - i.e

rm -f /home/ryan/projects/moose/petsc/linux-c-opt/lib/petsc/conf/pkg.conf.fblaslapack

and rerun configure.


Satish


From mbuerkle at web.de  Wed Feb 27 17:25:18 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Thu, 28 Feb 2019 00:25:18 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
	<99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
Message-ID: <trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>

Ok. I also tried without openmp which gave the same error. 
The segfault left aside, is MatAXPY supposed to work when Y is of type ELEMENTAL and X of type MATMPIAIJ ? It worked before but now it stops with mat elemental not supported.


> Von: "Stefano Zampini" <stefano.zampini at gmail.com>
> An: "Marius Buerkle" <mbuerkle at web.de>
> Cc: "Matthew Knepley" <knepley at gmail.com>, "PETSc users list" <petsc-users at mcs.anl.gov>
> Betreff: Re: [petsc-users] Nonconforming object sizes in axpy
>
> If you check ALL ERROR CODES by doing
> ierr = _AnyPetscCall_();CHKERRQ(ierr);
> The code will stop at the first error 
> 
> I see you are using openmp. Probably this is causing the problem, I don?t know what to say without a MWE.
> 
> 
> > On Feb 25, 2019, at 10:50 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> > 
> > <error_output.tar.gz>
> 
>

From stefano.zampini at gmail.com  Wed Feb 27 17:33:37 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 27 Feb 2019 15:33:37 -0800
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
	<99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
	<trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>
Message-ID: <7183BE65-3FA8-46BA-8DEA-612E82AD4878@gmail.com>



> On Feb 27, 2019, at 3:25 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> 
> Ok. I also tried without openmp which gave the same error. 
> The segfault left aside, is MatAXPY supposed to work when Y is of type ELEMENTAL and X of type MATMPIAIJ ? It worked before but now it stops with mat elemental not supported.
> 

I cannot find the error line that says Not supported for elemental.
The error PETSc is raising in the log you sent is  with respect to the nonconforming object sizes (which BTW are correct).
Could you please send the backtrace (without openmp) for the error regarding ?mat elemental not supported??



> 
>> Von: "Stefano Zampini" <stefano.zampini at gmail.com>
>> An: "Marius Buerkle" <mbuerkle at web.de>
>> Cc: "Matthew Knepley" <knepley at gmail.com>, "PETSc users list" <petsc-users at mcs.anl.gov>
>> Betreff: Re: [petsc-users] Nonconforming object sizes in axpy
>> 
>> If you check ALL ERROR CODES by doing
>> ierr = _AnyPetscCall_();CHKERRQ(ierr);
>> The code will stop at the first error 
>> 
>> I see you are using openmp. Probably this is causing the problem, I don?t know what to say without a MWE.
>> 
>> 
>>> On Feb 25, 2019, at 10:50 PM, Marius Buerkle <mbuerkle at web.de> wrote:
>>> 
>>> <error_output.tar.gz>
>> 
>> 


From jczhang at mcs.anl.gov  Wed Feb 27 17:42:26 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Wed, 27 Feb 2019 23:42:26 +0000
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
Message-ID: <CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>

Use srun <srun options> numactl -m 1 ./app OR srun <srun options> numactl -p 1 ./app
See bottom of https://www.nersc.gov/users/computational-systems/cori/configuration/knl-processor-modes/

--Junchao Zhang


On Wed, Feb 27, 2019 at 4:16 PM Sajid Ali via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi,

I ran a TS integrator for 25 steps on a Broadwell-Xeon and Xeon-Phi (KNL). The problem size is 5000x5000 and I'm using scalar=complex.

The program takes 125 seconds to run on Xeon and 451 seconds on KNL !

The first thing I want to change is to convert the memory access for the program on KNL from DRAM to MCDRAM. I did run the problem in an interactive SLURM job and specified -C quad,flat and yet I see DRAM is being used.

I'm attaching the PETSc log files and Intel APS reports as well. Any help on how I should change my runtime parameters on KNL will be highly appreciated. Thanks in advance.

--
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/29b4f01d/attachment-0001.html>

From mbuerkle at web.de  Wed Feb 27 17:59:12 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Thu, 28 Feb 2019 00:59:12 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <7183BE65-3FA8-46BA-8DEA-612E82AD4878@gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
	<99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
	<trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>
	<7183BE65-3FA8-46BA-8DEA-612E82AD4878@gmail.com>
Message-ID: <trinity-be8a5288-1b29-4aac-a004-929713d6add4-1551311952389@3c-app-webde-bap04>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/91925ff4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output.tar.gz
Type: application/octet-stream
Size: 1821 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/91925ff4/attachment.obj>

From stefano.zampini at gmail.com  Wed Feb 27 18:19:05 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 27 Feb 2019 16:19:05 -0800
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <trinity-be8a5288-1b29-4aac-a004-929713d6add4-1551311952389@3c-app-webde-bap04>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
	<99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
	<trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>
	<7183BE65-3FA8-46BA-8DEA-612E82AD4878@gmail.com>
	<trinity-be8a5288-1b29-4aac-a004-929713d6add4-1551311952389@3c-app-webde-bap04>
Message-ID: <366821B4-18D5-4305-8206-DA2CC59CA6E9@gmail.com>

I see the error now. You should pass SUBSET_NONZERO_PATTERN as MatStructure argument to MatAXPY
DIFFERENT_NONZERO_PATTERN now triggers preallocation, and MATELEMENTAL does not support MatGetRow.

> On Feb 27, 2019, at 3:59 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> 
> Here is the error message and please find an example attached. 
> thanks for the efforts !
>  
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type elemental
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1681-g73e1f9d014  GIT Date: 2019-02-24 19:52:00 +0100
> [0]PETSC ERROR: ./a.out on a  named tono-hpc1 by marius Thu Feb 28 08:56:48 2019
> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_dbg_with_slepc --with-fortran-kernels=0 --with-64-bit-indices=0 --CC=mpicc --COPTFLAGS="-g -O0 -qopenmp -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0 -qopenmp -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0 -qopenmp -traceback " --with-mpi=1 --with-x=0 --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251 --download-superlu_dist-cmake-arguments=-Denable_openmp=TRUE --download-hwloc=1 --download-sowing=1 --with-openmp=0 --with-pthread=0 --download-elemental=1 --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff --with-cxx-dialect=c++11 --with-scalar-type=complex --with-debugging=1 --with-valgrind=0 --with-blaslapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-valgrind=1
> [0]PETSC ERROR: #1 MatGetRow() line 570 in /home/marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 MatAXPY_Basic_Preallocate() line 95 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [0]PETSC ERROR: #3 MatAXPY() line 51 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: No support for this operation for this object type
> [1]PETSC ERROR: Mat type elemental
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1681-g73e1f9d014  GIT Date: 2019-02-24 19:52:00 +0100
> [1]PETSC ERROR: ./a.out on a  named tono-hpc1 by marius Thu Feb 28 08:56:48 2019
> [1]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_dbg_with_slepc --with-fortran-kernels=0 --with-64-bit-indices=0 --CC=mpicc --COPTFLAGS="-g -O0 -qopenmp -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0 -qopenmp -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0 -qopenmp -traceback " --with-mpi=1 --with-x=0 --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251 --download-superlu_dist-cmake-arguments=-Denable_openmp=TRUE --download-hwloc=1 --download-sowing=1 --with-openmp=0 --with-pthread=0 --download-elemental=1 --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff --with-cxx-dialect=c++11 --with-scalar-type=complex --with-debugging=1 --with-valgrind=0 --with-blaslapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-valgrind=1
> [1]PETSC ERROR: #1 MatGetRow() line 570 in /home/marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> [1]PETSC ERROR: #2 MatAXPY_Basic_Preallocate() line 95 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [1]PETSC ERROR: #3 MatAXPY() line 51 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: No support for this operation for this object type
> [2]PETSC ERROR: Mat type elemental
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [2]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1681-g73e1f9d014  GIT Date: 2019-02-24 19:52:00 +0100
> [2]PETSC ERROR: ./a.out on a  named tono-hpc1 by marius Thu Feb 28 08:56:48 2019
> [2]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_dbg_with_slepc --with-fortran-kernels=0 --with-64-bit-indices=0 --CC=mpicc --COPTFLAGS="-g -O0 -qopenmp -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0 -qopenmp -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0 -qopenmp -traceback " --with-mpi=1 --with-x=0 --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251 --download-superlu_dist-cmake-arguments=-Denable_openmp=TRUE --download-hwloc=1 --download-sowing=1 --with-openmp=0 --with-pthread=0 --download-elemental=1 --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff --with-cxx-dialect=c++11 --with-scalar-type=complex --with-debugging=1 --with-valgrind=0 --with-blaslapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-valgrind=1
> [2]PETSC ERROR: #1 MatGetRow() line 570 in /home/marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> [2]PETSC ERROR: #2 MatAXPY_Basic_Preallocate() line 95 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [2]PETSC ERROR: #3 MatAXPY() line 51 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [3]PETSC ERROR: No support for this operation for this object type
> [3]PETSC ERROR: Mat type elemental
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [3]PETSC ERROR: Petsc Development GIT revision: v3.10.3-1681-g73e1f9d014  GIT Date: 2019-02-24 19:52:00 +0100
> [3]PETSC ERROR: ./a.out on a  named tono-hpc1 by marius Thu Feb 28 08:56:48 2019
> [3]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/intel_openmpi_openmp_mkl_dbg_with_slepc --with-fortran-kernels=0 --with-64-bit-indices=0 --CC=mpicc --COPTFLAGS="-g -O0 -qopenmp -std=c11" --CXX=mpicxx --CXXOPTFLAGS="-g -O0 -qopenmp -std=c++14" --FC=mpif90 --FOPTFLAGS="-g -O0 -qopenmp -traceback " --with-mpi=1 --with-x=0 --download-parmetis=1 --download-metis=1 --download-superlu_dist=1 --download-superlu_dist-commit=93a9aa53142253f2a56891e09c6f950d250ae251 --download-superlu_dist-cmake-arguments=-Denable_openmp=TRUE --download-hwloc=1 --download-sowing=1 --with-openmp=0 --with-pthread=0 --download-elemental=1 --download-elemental-commit=de7b5bea1abf5f626b91582f742cf99e2e551bff --with-cxx-dialect=c++11 --with-scalar-type=complex --with-debugging=1 --with-valgrind=0 --with-blaslapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-scalapack-lib=" -L/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -liomp5 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_cpardiso-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-mkl_sparse_optimize-dir=/home/marius/intel/compilers_and_libraries_2019.1.144/linux/mkl --with-valgrind=1
> [3]PETSC ERROR: #1 MatGetRow() line 570 in /home/marius/prog/petsc/git/petsc/src/mat/interface/matrix.c
> [3]PETSC ERROR: #2 MatAXPY_Basic_Preallocate() line 95 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
> [3]PETSC ERROR: #3 MatAXPY() line 51 in /home/marius/prog/petsc/git/petsc/src/mat/utils/axpy.c
>  
> 
> Von: "Stefano Zampini" <stefano.zampini at gmail.com>
> An: "Marius Buerkle" <mbuerkle at web.de>
> Cc: "Matthew Knepley" <knepley at gmail.com>, "PETSc users list" <petsc-users at mcs.anl.gov>
> Betreff: Re: [petsc-users] Nonconforming object sizes in axpy
> 
> 
> > On Feb 27, 2019, at 3:25 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> >
> > Ok. I also tried without openmp which gave the same error.
> > The segfault left aside, is MatAXPY supposed to work when Y is of type ELEMENTAL and X of type MATMPIAIJ ? It worked before but now it stops with mat elemental not supported.
> >
> 
> I cannot find the error line that says Not supported for elemental.
> The error PETSc is raising in the log you sent is with respect to the nonconforming object sizes (which BTW are correct).
> Could you please send the backtrace (without openmp) for the error regarding ?mat elemental not supported??
> 
> 
> 
> >
> >> Von: "Stefano Zampini" <stefano.zampini at gmail.com>
> >> An: "Marius Buerkle" <mbuerkle at web.de>
> >> Cc: "Matthew Knepley" <knepley at gmail.com>, "PETSc users list" <petsc-users at mcs.anl.gov>
> >> Betreff: Re: [petsc-users] Nonconforming object sizes in axpy
> >>
> >> If you check ALL ERROR CODES by doing
> >> ierr = _AnyPetscCall_();CHKERRQ(ierr);
> >> The code will stop at the first error
> >>
> >> I see you are using openmp. Probably this is causing the problem, I don?t know what to say without a MWE.
> >>
> >>
> >>> On Feb 25, 2019, at 10:50 PM, Marius Buerkle <mbuerkle at web.de> wrote:
> >>>
> >>> <error_output.tar.gz>
> >>
> >>
>  
> <output.tar.gz>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/246a419a/attachment-0001.html>

From panagiotis.dafnakis at polito.it  Wed Feb 27 18:01:17 2019
From: panagiotis.dafnakis at polito.it (DAFNAKIS PANAGIOTIS)
Date: Thu, 28 Feb 2019 01:01:17 +0100
Subject: [petsc-users] (no subject)
Message-ID: <web-19704499@backmail1.polito.it>

Hi everybody,

I am trying to install PETSc version 3.10.3 on OSX Sierra 
10.13.6 with the following configure options
./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 
--PETSC_ARCH=sierra-dbg --with-debugging=1 
--download-hypre=1 --with-x=0

however I am getting the following error messages when I 
do 'make check'. See below the resulting message. Any 
suggestions?

Thanks,

--Panos

panos at Sierra-iMac:~/Softwares/PETSc-Bitbucket/PETSc$ make 
PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc 
PETSC_ARCH=sierra-dbg check
Running test examples to verify correct installation
Using 
PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc and 
PETSC_ARCH=sierra-dbg
make[2]: [ex19.PETSc] Error 2 (ignored)
*******************Error detected during compile or 
link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials 
ex19
*********************************************************************************
mpicc -o ex19.o -c -Wall -Wwrite-strings 
-Wno-strict-aliasing -Wno-unknown-pragmas 
-Qunused-arguments -fvisibility=hidden -g3 
  -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include 
-I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include 
   `pwd`/ex19.c
mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
-Wl,suppress -Wl,-commons,use_dylibs 
-Wl,-search_paths_first -Wl,-no_compact_unwind    -Wall 
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas 
-Qunused-arguments -fvisibility=hidden -g3  -o ex19 ex19.o 
 -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib 
-L/usr/local/Cellar/mpich/3.3/lib 
-Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 
-L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 
-Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 
-L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 
-Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries 
-L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries 
-Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources 
-L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources 
-Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources 
-L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources 
-lpetsc -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran 
-lquadmath -lm -lGFXShared -lGLU -lGL -lGLImage 
-lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG 
-lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd 
and library file 
/System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd 
and library file 
/System/Library/Frameworks//IOKit.framework/Versions/A/IOKit 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd 
and library file 
/System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd 
and library file 
/System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd 
and library file 
/System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//Security.framework/Versions/A/Security.tbd 
and library file 
/System/Library/Frameworks//Security.framework/Versions/A/Security 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd 
and library file 
/System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd 
and library file 
/System/Library/Frameworks//Foundation.framework/Versions/C/Foundation 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd 
and library file 
/System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd 
and library file 
/System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd 
and library file 
/System/Library/Frameworks//CoreText.framework/Versions/A/CoreText 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd 
and library file 
/System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd 
and library file 
/System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd 
and library file 
/System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd 
and library file 
/System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd 
and library file 
/System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd 
and library file 
/System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd 
and library file 
/System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd 
and library file 
/System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd 
and library file 
/System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS 
are out of sync. Falling back to library file for linking.
Undefined symbols for architecture x86_64:
   "_jpeg_CreateCompress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_destroy_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_finish_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_set_defaults", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_start_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_std_error", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_stdio_dest", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_write_scanlines", referenced from:
       import-atom in libpetsc.dylib
   "_png_create_info_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_create_write_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_destroy_write_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_init_io", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_IHDR", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_PLTE", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_longjmp_fn", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_end", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_info", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_row", referenced from:
       import-atom in libpetsc.dylib
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use 
-v to see invocation)
make[3]: [ex19] Error 1 (ignored)
/usr/bin/dsymutil ex19
error: cannot parse the debug map for 'ex19': No such file 
or directory
make[3]: *** [ex19] Error 1
rm ex19.o
1,5c1,2
< lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
<   0 SNES Function norm 0.0406612
<   1 SNES Function norm 4.12227e-06
<   2 SNES Function norm 6.098e-11
< Number of SNES iterations = 2
---
> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or directory)
> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or directory)
/Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials
Possible problem with ex19_hypre, diffs above
=========================================
*******************Error detected during compile or 
link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials 
ex5f
*********************************************************
mpif90 -c -Wall -ffree-line-length-0 
-Wno-unused-dummy-argument -g 
   -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include 
-I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include 
   -o ex5f.o ex5f.F90
mpif90 -Wl,-multiply_defined,suppress 
-Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs 
-Wl,-search_paths_first -Wl,-no_compact_unwind   -Wall 
-ffree-line-length-0 -Wno-unused-dummy-argument -g   -o 
ex5f ex5f.o 
 -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib 
-Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib 
-L/usr/local/Cellar/mpich/3.3/lib 
-Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 
-L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 
-Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 
-L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 
-Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries 
-L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries 
-Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources 
-L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources 
-Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources 
-L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources 
-lpetsc -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran 
-lquadmath -lm -lGFXShared -lGLU -lGL -lGLImage 
-lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG 
-lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd 
and library file 
/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd 
and library file 
/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd 
and library file 
/System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd 
and library file 
/System/Library/Frameworks//IOKit.framework/Versions/A/IOKit 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd 
and library file 
/System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd 
and library file 
/System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd 
and library file 
/System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//Security.framework/Versions/A/Security.tbd 
and library file 
/System/Library/Frameworks//Security.framework/Versions/A/Security 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd 
and library file 
/System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd 
and library file 
/System/Library/Frameworks//Foundation.framework/Versions/C/Foundation 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd 
and library file 
/System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd 
and library file 
/System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd 
and library file 
/System/Library/Frameworks//CoreText.framework/Versions/A/CoreText 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd 
and library file 
/System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd 
and library file 
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd 
and library file 
/System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd 
and library file 
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd 
and library file 
/System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd 
and library file 
/System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd 
and library file 
/System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd 
and library file 
/System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd 
and library file 
/System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd 
and library file 
/System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement 
are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file 
/System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd 
and library file 
/System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS 
are out of sync. Falling back to library file for linking.
Undefined symbols for architecture x86_64:
   "_jpeg_CreateCompress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_destroy_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_finish_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_set_defaults", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_start_compress", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_std_error", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_stdio_dest", referenced from:
       import-atom in libpetsc.dylib
   "_jpeg_write_scanlines", referenced from:
       import-atom in libpetsc.dylib
   "_png_create_info_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_create_write_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_destroy_write_struct", referenced from:
       import-atom in libpetsc.dylib
   "_png_init_io", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_IHDR", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_PLTE", referenced from:
       import-atom in libpetsc.dylib
   "_png_set_longjmp_fn", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_end", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_info", referenced from:
       import-atom in libpetsc.dylib
   "_png_write_row", referenced from:
       import-atom in libpetsc.dylib
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[3]: [ex5f] Error 1 (ignored)
rm ex5f.o
Completed test examples

From stefano.zampini at gmail.com  Wed Feb 27 18:22:44 2019
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 27 Feb 2019 16:22:44 -0800
Subject: [petsc-users] (no subject)
In-Reply-To: <web-19704499@backmail1.polito.it>
References: <web-19704499@backmail1.polito.it>
Message-ID: <240721E9-7A53-4DF6-B846-904ADB84D97D@gmail.com>

This is how I fixed this. You should configure PETSc using these libraries from brew.

    '--with-giflib-dir=/usr/local',
    '--with-libjpeg-dir=/usr/local',
    '--with-libpng-dir=/usr/local',
    '--with-opengl-dir=/usr/X11',
    '--with-zlib-dir=/usr/local/opt/zlib',

> On Feb 27, 2019, at 4:01 PM, DAFNAKIS PANAGIOTIS via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi everybody,
> 
> I am trying to install PETSc version 3.10.3 on OSX Sierra 10.13.6 with the following configure options
> ./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=sierra-dbg --with-debugging=1 --download-hypre=1 --with-x=0
> 
> however I am getting the following error messages when I do 'make check'. See below the resulting message. Any suggestions?
> 
> Thanks,
> 
> --Panos
> 
> panos at Sierra-iMac:~/Softwares/PETSc-Bitbucket/PETSc$ make PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc PETSC_ARCH=sierra-dbg check
> Running test examples to verify correct installation
> Using PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc and PETSC_ARCH=sierra-dbg
> make[2]: [ex19.PETSc] Error 2 (ignored)
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex19
> *********************************************************************************
> mpicc -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Qunused-arguments -fvisibility=hidden -g3  -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include   `pwd`/ex19.c
> mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Qunused-arguments -fvisibility=hidden -g3  -o ex19 ex19.o -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd and library file /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd and library file /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd and library file /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and library file /System/Library/Frameworks//Security.framework/Versions/A/Security are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd and library file /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and library file /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and library file /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and library file /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and library file /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd and library file /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and library file /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd and library file /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd and library file /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd and library file /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS are out of sync. Falling back to library file for linking.
> Undefined symbols for architecture x86_64:
>  "_jpeg_CreateCompress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_destroy_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_finish_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_set_defaults", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_start_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_std_error", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_stdio_dest", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_write_scanlines", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_create_info_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_create_write_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_destroy_write_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_init_io", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_IHDR", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_PLTE", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_longjmp_fn", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_end", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_info", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_row", referenced from:
>      import-atom in libpetsc.dylib
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> make[3]: [ex19] Error 1 (ignored)
> /usr/bin/dsymutil ex19
> error: cannot parse the debug map for 'ex19': No such file or directory
> make[3]: *** [ex19] Error 1
> rm ex19.o
> 1,5c1,2
> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> <   0 SNES Function norm 0.0406612
> <   1 SNES Function norm 4.12227e-06
> <   2 SNES Function norm 6.098e-11
> < Number of SNES iterations = 2
> ---
>> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or directory)
>> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or directory)
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials
> Possible problem with ex19_hypre, diffs above
> =========================================
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex5f
> *********************************************************
> mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g   -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include   -o ex5f.o ex5f.F90
> mpif90 -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind   -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g   -o ex5f ex5f.o -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0 -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd and library file /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd and library file /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd and library file /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd and library file /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd and library file /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and library file /System/Library/Frameworks//Security.framework/Versions/A/Security are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd and library file /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and library file /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and library file /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and library file /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd and library file /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and library file /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd and library file /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd and library file /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and library file /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd and library file /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd and library file /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd and library file /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS are out of sync. Falling back to library file for linking.
> Undefined symbols for architecture x86_64:
>  "_jpeg_CreateCompress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_destroy_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_finish_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_set_defaults", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_start_compress", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_std_error", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_stdio_dest", referenced from:
>      import-atom in libpetsc.dylib
>  "_jpeg_write_scanlines", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_create_info_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_create_write_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_destroy_write_struct", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_init_io", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_IHDR", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_PLTE", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_set_longjmp_fn", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_end", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_info", referenced from:
>      import-atom in libpetsc.dylib
>  "_png_write_row", referenced from:
>      import-atom in libpetsc.dylib
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status
> make[3]: [ex5f] Error 1 (ignored)
> rm ex5f.o
> Completed test examples

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/d101ea47/attachment-0001.html>

From balay at mcs.anl.gov  Wed Feb 27 18:27:39 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 28 Feb 2019 00:27:39 +0000
Subject: [petsc-users] (no subject)
In-Reply-To: <web-19704499@backmail1.polito.it>
References: <web-19704499@backmail1.polito.it>
Message-ID: <alpine.LFD.2.21.1902271825400.2786@sb>

<removing petsc-dev>

Can you send configure.log from this build?

Satish

On Thu, 28 Feb 2019, DAFNAKIS PANAGIOTIS via petsc-users wrote:

> Hi everybody,
> 
> I am trying to install PETSc version 3.10.3 on OSX Sierra 10.13.6 with the
> following configure options
> ./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=sierra-dbg
> --with-debugging=1 --download-hypre=1 --with-x=0
> 
> however I am getting the following error messages when I do 'make check'. See
> below the resulting message. Any suggestions?
> 
> Thanks,
> 
> --Panos
> 
> panos at Sierra-iMac:~/Softwares/PETSc-Bitbucket/PETSc$ make
> PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc PETSC_ARCH=sierra-dbg
> check
> Running test examples to verify correct installation
> Using PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc and
> PETSC_ARCH=sierra-dbg
> make[2]: [ex19.PETSc] Error 2 (ignored)
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex19
> *********************************************************************************
> mpicc -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -Qunused-arguments -fvisibility=hidden -g3
> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include
> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include
> `pwd`/ex19.c
> mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress
> -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind
> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -Qunused-arguments -fvisibility=hidden -g3  -o ex19 ex19.o
> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib
> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
> -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
> -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
> -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
> -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
> -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources
> -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc
> -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared
> -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG
> -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and
> library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd
> and library file
> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library
> file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd
> and library file
> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd
> and library file
> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd
> and library file
> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and
> library file
> /System/Library/Frameworks//Security.framework/Versions/A/Security are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd
> and library file
> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and
> library file
> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library
> file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and
> library file
> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and
> library file
> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and
> library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and
> library file
> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd
> and library file
> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library
> file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and
> library file
> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd
> and library file
> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd
> and library file
> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd
> and library file
> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and
> library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
> are out of sync. Falling back to library file for linking.
> Undefined symbols for architecture x86_64:
>   "_jpeg_CreateCompress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_destroy_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_finish_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_set_defaults", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_start_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_std_error", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_stdio_dest", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_write_scanlines", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_create_info_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_create_write_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_destroy_write_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_init_io", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_IHDR", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_PLTE", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_longjmp_fn", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_end", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_info", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_row", referenced from:
>       import-atom in libpetsc.dylib
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see
> invocation)
> make[3]: [ex19] Error 1 (ignored)
> /usr/bin/dsymutil ex19
> error: cannot parse the debug map for 'ex19': No such file or directory
> make[3]: *** [ex19] Error 1
> rm ex19.o
> 1,5c1,2
> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> <   0 SNES Function norm 0.0406612
> <   1 SNES Function norm 4.12227e-06
> <   2 SNES Function norm 6.098e-11
> < Number of SNES iterations = 2
> ---
> > [proxy:0:0 at Sierra-iMac.local] HYDU_create_process
> > (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or
> > directory)
> > [proxy:0:0 at Sierra-iMac.local] HYDU_create_process
> > (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or
> > directory)
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials
> Possible problem with ex19_hypre, diffs above
> =========================================
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex5f
> *********************************************************
> mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g
> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include
> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include   -o ex5f.o
> ex5f.F90
> mpif90 -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress
> -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind   -Wall
> -ffree-line-length-0 -Wno-unused-dummy-argument -g   -o ex5f ex5f.o
> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
> -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib
> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
> -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
> -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
> -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
> -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
> -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources
> -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc
> -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared
> -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG
> -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and
> library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd
> and library file
> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd
> and library file
> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd
> and library file
> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library
> file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd
> and library file
> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd
> and library file
> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd
> and library file
> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and
> library file
> /System/Library/Frameworks//Security.framework/Versions/A/Security are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd
> and library file
> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and
> library file
> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library
> file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and
> library file
> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and
> library file
> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and
> library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd
> and library file
> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and
> library file
> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd
> and library file
> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd
> and library file
> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library
> file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of
> sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and
> library file
> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out
> of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd
> and library file
> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are
> out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd
> and library file
> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd
> and library file
> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement
> are out of sync. Falling back to library file for linking.
> ld: warning: text-based stub file
> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and
> library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
> are out of sync. Falling back to library file for linking.
> Undefined symbols for architecture x86_64:
>   "_jpeg_CreateCompress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_destroy_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_finish_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_set_defaults", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_start_compress", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_std_error", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_stdio_dest", referenced from:
>       import-atom in libpetsc.dylib
>   "_jpeg_write_scanlines", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_create_info_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_create_write_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_destroy_write_struct", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_init_io", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_IHDR", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_PLTE", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_set_longjmp_fn", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_end", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_info", referenced from:
>       import-atom in libpetsc.dylib
>   "_png_write_row", referenced from:
>       import-atom in libpetsc.dylib
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status
> make[3]: [ex5f] Error 1 (ignored)
> rm ex5f.o
> Completed test examples
> 


From asbhalla at sdsu.edu  Wed Feb 27 18:50:34 2019
From: asbhalla at sdsu.edu (Amneet Pal Bhalla)
Date: Wed, 27 Feb 2019 16:50:34 -0800
Subject: [petsc-users] Error with petsc on OSX with libpng
In-Reply-To: <alpine.LFD.2.21.1902271825400.2786@sb>
References: <web-19704499@backmail1.polito.it>
	<alpine.LFD.2.21.1902271825400.2786@sb>
Message-ID: <E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>

A non-text attachment was scrubbed...
Name: configure_petsc.log
Type: application/octet-stream
Size: 6322584 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/599f3cd7/attachment-0001.obj>
-------------- next part --------------


> On Feb 27, 2019, at 4:27 PM, Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> <removing petsc-dev>
> 
> Can you send configure.log from this build?
> 
> Satish
> 
> On Thu, 28 Feb 2019, DAFNAKIS PANAGIOTIS via petsc-users wrote:
> 
>> Hi everybody,
>> 
>> I am trying to install PETSc version 3.10.3 on OSX Sierra 10.13.6 with the
>> following configure options
>> ./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=sierra-dbg
>> --with-debugging=1 --download-hypre=1 --with-x=0
>> 
>> however I am getting the following error messages when I do 'make check'. See
>> below the resulting message. Any suggestions?
>> 
>> Thanks,
>> 
>> --Panos
>> 
>> panos at Sierra-iMac:~/Softwares/PETSc-Bitbucket/PETSc$ make
>> PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc PETSC_ARCH=sierra-dbg
>> check
>> Running test examples to verify correct installation
>> Using PETSC_DIR=/Users/panos/Softwares/PETSc-Bitbucket/PETSc and
>> PETSC_ARCH=sierra-dbg
>> make[2]: [ex19.PETSc] Error 2 (ignored)
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex19
>> *********************************************************************************
>> mpicc -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
>> -Wno-unknown-pragmas -Qunused-arguments -fvisibility=hidden -g3
>> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include
>> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include
>> `pwd`/ex19.c
>> mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress
>> -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind
>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -Qunused-arguments -fvisibility=hidden -g3  -o ex19 ex19.o
>> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib
>> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
>> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
>> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
>> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
>> -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
>> -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
>> -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
>> -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
>> -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources
>> -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc
>> -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared
>> -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG
>> -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and
>> library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd
>> and library file
>> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library
>> file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd
>> and library file
>> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd
>> and library file
>> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd
>> and library file
>> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and
>> library file
>> /System/Library/Frameworks//Security.framework/Versions/A/Security are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd
>> and library file
>> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and
>> library file
>> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library
>> file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and
>> library file
>> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and
>> library file
>> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and
>> library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and
>> library file
>> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd
>> and library file
>> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library
>> file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and
>> library file
>> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd
>> and library file
>> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd
>> and library file
>> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd
>> and library file
>> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and
>> library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
>> are out of sync. Falling back to library file for linking.
>> Undefined symbols for architecture x86_64:
>>  "_jpeg_CreateCompress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_destroy_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_finish_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_set_defaults", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_start_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_std_error", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_stdio_dest", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_write_scanlines", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_create_info_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_create_write_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_destroy_write_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_init_io", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_IHDR", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_PLTE", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_longjmp_fn", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_end", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_info", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_row", referenced from:
>>      import-atom in libpetsc.dylib
>> ld: symbol(s) not found for architecture x86_64
>> clang: error: linker command failed with exit code 1 (use -v to see
>> invocation)
>> make[3]: [ex19] Error 1 (ignored)
>> /usr/bin/dsymutil ex19
>> error: cannot parse the debug map for 'ex19': No such file or directory
>> make[3]: *** [ex19] Error 1
>> rm ex19.o
>> 1,5c1,2
>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
>> <   0 SNES Function norm 0.0406612
>> <   1 SNES Function norm 4.12227e-06
>> <   2 SNES Function norm 6.098e-11
>> < Number of SNES iterations = 2
>> ---
>>> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process
>>> (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or
>>> directory)
>>> [proxy:0:0 at Sierra-iMac.local] HYDU_create_process
>>> (utils/launch/launch.c:74): execvp error on file ./ex19 (No such file or
>>> directory)
>> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials
>> Possible problem with ex19_hypre, diffs above
>> =========================================
>> *******************Error detected during compile or link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /Users/panos/Softwares/PETSc-Bitbucket/PETSc/src/snes/examples/tutorials ex5f
>> *********************************************************
>> mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g
>> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/include
>> -I/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/include   -o ex5f.o
>> ex5f.F90
>> mpif90 -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress
>> -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind   -Wall
>> -ffree-line-length-0 -Wno-unused-dummy-argument -g   -o ex5f ex5f.o
>> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -Wl,-rpath,/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -L/Users/panos/Softwares/PETSc-Bitbucket/PETSc/sierra-dbg/lib
>> -Wl,-rpath,/usr/local/Cellar/mpich/3.3/lib -L/usr/local/Cellar/mpich/3.3/lib
>> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
>> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8/gcc/x86_64-apple-darwin17.7.0/8.3.0
>> -Wl,-rpath,/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
>> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8
>> -Wl,-rpath,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
>> -L/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries
>> -Wl,-rpath,/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
>> -L/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources
>> -Wl,-rpath,/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources
>> -L/System/Library/Frameworks/ImageIO.framework/Versions/A/Resources -lpetsc
>> -lHYPRE -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lGFXShared
>> -lGLU -lGL -lGLImage -lCVMSPluginSupport -lFontParser -lFontRegistry -lJPEG
>> -lTIFF -lPng -lGIF -lJP2 -lRadiance -lCoreVMClient -ldl
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.tbd and
>> library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.tbd
>> and library file
>> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.tbd
>> and library file
>> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation.tbd
>> and library file
>> /System/Library/Frameworks//CoreFoundation.framework/Versions/A/CoreFoundation
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit.tbd and library
>> file /System/Library/Frameworks//IOKit.framework/Versions/A/IOKit are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator.tbd
>> and library file
>> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices.tbd
>> and library file
>> /System/Library/Frameworks//ApplicationServices.framework/Versions/A/ApplicationServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics.tbd
>> and library file
>> /System/Library/Frameworks//CoreGraphics.framework/Versions/A/CoreGraphics are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//Security.framework/Versions/A/Security.tbd and
>> library file
>> /System/Library/Frameworks//Security.framework/Versions/A/Security are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices.tbd
>> and library file
>> /System/Library/Frameworks//CoreServices.framework/Versions/A/CoreServices are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation.tbd and
>> library file
>> /System/Library/Frameworks//Foundation.framework/Versions/C/Foundation are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library
>> file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface.tbd and
>> library file
>> /System/Library/Frameworks//IOSurface.framework/Versions/A/IOSurface are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText.tbd and
>> library file
>> /System/Library/Frameworks//CoreText.framework/Versions/A/CoreText are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO.tbd and
>> library file /System/Library/Frameworks//ImageIO.framework/Versions/A/ImageIO
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis.tbd
>> and library file
>> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork.tbd and
>> library file
>> /System/Library/Frameworks//CFNetwork.framework/Versions/A/CFNetwork are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList.tbd
>> and library file
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration.tbd
>> and library file
>> /System/Library/Frameworks//DiskArbitration.framework/Versions/A/DiskArbitration
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL.tbd and library
>> file /System/Library/Frameworks//OpenGL.framework/Versions/A/OpenGL are out of
>> sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio.tbd and
>> library file
>> /System/Library/Frameworks//CoreAudio.framework/Versions/A/CoreAudio are out
>> of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox.tbd
>> and library file
>> /System/Library/Frameworks//AudioToolbox.framework/Versions/A/AudioToolbox are
>> out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory.tbd
>> and library file
>> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement.tbd
>> and library file
>> /System/Library/Frameworks//ServiceManagement.framework/Versions/A/ServiceManagement
>> are out of sync. Falling back to library file for linking.
>> ld: warning: text-based stub file
>> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS.tbd and
>> library file /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
>> are out of sync. Falling back to library file for linking.
>> Undefined symbols for architecture x86_64:
>>  "_jpeg_CreateCompress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_destroy_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_finish_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_set_defaults", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_start_compress", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_std_error", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_stdio_dest", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_jpeg_write_scanlines", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_create_info_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_create_write_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_destroy_write_struct", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_init_io", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_IHDR", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_PLTE", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_set_longjmp_fn", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_end", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_info", referenced from:
>>      import-atom in libpetsc.dylib
>>  "_png_write_row", referenced from:
>>      import-atom in libpetsc.dylib
>> ld: symbol(s) not found for architecture x86_64
>> collect2: error: ld returned 1 exit status
>> make[3]: [ex5f] Error 1 (ignored)
>> rm ex5f.o
>> Completed test examples
>> 
> 


From sajidsyed2021 at u.northwestern.edu  Wed Feb 27 19:02:43 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Wed, 27 Feb 2019 19:02:43 -0600
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
	<CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
Message-ID: <CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>

Hi Junchao,

I?m confused with the syntax. If I submit the following as my job script, I
get an error :

#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00

module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify
-ts_type cn -prop_steps 25 -pc_type gamg -ts_monitor -log_view

The error is :
srun: cluster configuration lacks support for cpu binding
srun: error: Unable to create step for job 916208: More processors
requested than permitted

I?m following the advice as given at slide 33 of
https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf

For further info, I?m using LCRC at ANL.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/88f81bce/attachment.html>

From balay at mcs.anl.gov  Wed Feb 27 19:12:50 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 28 Feb 2019 01:12:50 +0000
Subject: [petsc-users] Error with petsc on OSX with libpng
In-Reply-To: <E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>
References: <web-19704499@backmail1.polito.it>
	<alpine.LFD.2.21.1902271825400.2786@sb>
	<E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>
Message-ID: <alpine.LFD.2.21.1902271908210.8544@sb>

Can you try the following and see if it makes a difference?

./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=sierra-dbg --with-debugging=1 --download-hypre=1 --with-x=0 LIBS='-L/usr/local/Cellar/mpich/3.3/lib -lmpifort -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -lgfortran'

For some reason your xcode compiler is giving these warnings during linktime - and I don't know why. These warnings are confusing petsc configure.

>>>>>>>
ld: warning: text-based stub file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of sync. Falling back to library file for linking.
<<<<<

Satish

On Wed, 27 Feb 2019, Amneet Pal Bhalla via petsc-users wrote:


  [NON-Text Body part not included]



From asbhalla at sdsu.edu  Wed Feb 27 19:14:47 2019
From: asbhalla at sdsu.edu (Amneet Pal Bhalla)
Date: Wed, 27 Feb 2019 17:14:47 -0800
Subject: [petsc-users] Error with petsc on OSX with libpng
In-Reply-To: <alpine.LFD.2.21.1902271908210.8544@sb>
References: <web-19704499@backmail1.polito.it>
	<alpine.LFD.2.21.1902271825400.2786@sb>
	<E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>
	<alpine.LFD.2.21.1902271908210.8544@sb>
Message-ID: <CA+qmB1XG48+cjj-q8PW-P+GVboYtMuCHhROuQ+OwLpG+ZNRFdg@mail.gmail.com>

Ah, I see. I will try to make sure that my Xcode is working. Did you try to
update XCode yet.

On Wed, Feb 27, 2019 at 5:12 PM Balay, Satish <balay at mcs.anl.gov> wrote:

> Can you try the following and see if it makes a difference?
>
> ./configure --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=sierra-dbg
> --with-debugging=1 --download-hypre=1 --with-x=0
> LIBS='-L/usr/local/Cellar/mpich/3.3/lib -lmpifort
> -L/usr/local/Cellar/gcc/8.3.0/lib/gcc/8 -lgfortran'
>
> For some reason your xcode compiler is giving these warnings during
> linktime - and I don't know why. These warnings are confusing petsc
> configure.
>
> >>>>>>>
> ld: warning: text-based stub file
> /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and
> library file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL
> are out of sync. Falling back to library file for linking.
> <<<<<
>
> Satish
>
> On Wed, 27 Feb 2019, Amneet Pal Bhalla via petsc-users wrote:
>
>
>   [NON-Text Body part not included]
>
>
> --
---
Amneet Bhalla
Assistant Professor, Department of Mechanical Engineering
San Diego State University
5500 Campanile Drive
San Diego, CA 92182 USA
Web: https://bhallagroup.wixsite.com/cfdlab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190227/c8b1c3e2/attachment.html>

From balay at mcs.anl.gov  Wed Feb 27 19:20:13 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 28 Feb 2019 01:20:13 +0000
Subject: [petsc-users] Error with petsc on OSX with libpng
In-Reply-To: <alpine.LFD.2.21.1902271908210.8544@sb>
References: <web-19704499@backmail1.polito.it>
	<alpine.LFD.2.21.1902271825400.2786@sb>
	<E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>
	<alpine.LFD.2.21.1902271908210.8544@sb>
Message-ID: <alpine.LFD.2.21.1902271917130.8544@sb>

On Thu, 28 Feb 2019, Balay, Satish via petsc-users wrote:

> For some reason your xcode compiler is giving these warnings during linktime - and I don't know why. These warnings are confusing petsc configure.
> 
> >>>>>>>
> ld: warning: text-based stub file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL.tbd and library file /System/Library/Frameworks//OpenCL.framework/Versions/A/OpenCL are out of sync. Falling back to library file for linking.
> <<<<<

Here are some potential fixes you can try..

https://stackoverflow.com/questions/51314888/ld-warning-text-based-stub-file-are-out-of-sync-falling-back-to-library-file#51619171
https://forums.developer.apple.com/thread/97850

>>>>>>>>>>>
This is due to an out-of-sync Xcode/command-line tools issue. Reinstalling the command-line tools seems to fix the issue per http://sd.jtimothyking.com/2018/07/26/stub-file-and-library-file-out-of-sync/:

 

$ sudo mv /Library/Developer/CommandLineTools /Library/Developer/CommandLineTools.old

$ xcode-select --install

$ sudo rm -rf /Library/Developer/CommandLineTools.old
<<<<<<<<<<<

Satish

From balay at mcs.anl.gov  Wed Feb 27 19:32:46 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 28 Feb 2019 01:32:46 +0000
Subject: [petsc-users] Error with petsc on OSX with libpng
In-Reply-To: <CA+qmB1XG48+cjj-q8PW-P+GVboYtMuCHhROuQ+OwLpG+ZNRFdg@mail.gmail.com>
References: <web-19704499@backmail1.polito.it>
	<alpine.LFD.2.21.1902271825400.2786@sb>
	<E57CD43F-B4B6-451D-BB38-B76A3937B5C8@sdsu.edu>
	<alpine.LFD.2.21.1902271908210.8544@sb>
	<CA+qmB1XG48+cjj-q8PW-P+GVboYtMuCHhROuQ+OwLpG+ZNRFdg@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1902271923150.8544@sb>

On Wed, 27 Feb 2019, Amneet Pal Bhalla via petsc-users wrote:

> Ah, I see. I will try to make sure that my Xcode is working. Did you try to
> update XCode yet.

Hm - currently petsc is tested with:

>>>>
balay at ipro^~ $ xcodebuild -version
Xcode 10.1
Build version 10B61
balay at ipro^~ $ clang --version
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
<<<<<<<

Your compiler appears to be a bit older.

>>>
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
<<<<

I don't see any such issues.. [Its likely I don't have command line tools installed - so don't get into this issue]

Satish

From yyang85 at stanford.edu  Wed Feb 27 19:33:58 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 28 Feb 2019 01:33:58 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
Message-ID: <DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>

Thanks, I fixed that, but I?m not actually calling the testScatters() function in my implementation (in the constructor, the only functions I called are setFields and setScatters). So the problem couldn?t have been that?

Best,
Yuyun

From: Zhang, Junchao <jczhang at mcs.anl.gov>
Sent: Wednesday, February 27, 2019 10:50 AM
To: Yuyun Yang <yyang85 at stanford.edu>
Cc: Matthew Knepley <knepley at gmail.com>; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/b754d908/attachment-0001.html>

From jczhang at mcs.anl.gov  Wed Feb 27 20:22:40 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 28 Feb 2019 02:22:40 +0000
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
	<CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
	<CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
Message-ID: <CA+MQGp9wOYpp6LpJSSWKNTPqc0C82OG2O075Bkk+6XkLgxo=yQ@mail.gmail.com>




On Wed, Feb 27, 2019 at 7:03 PM Sajid Ali <sajidsyed2021 at u.northwestern.edu<mailto:sajidsyed2021 at u.northwestern.edu>> wrote:

Hi Junchao,

I?m confused with the syntax. If I submit the following as my job script, I get an error :

#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00

module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify -ts_type cn -prop_steps 25 -pc_type gamg -ts_monitor -log_view


The error is :
srun: cluster configuration lacks support for cpu binding

This cluster does not support cpu binding.  You need to remove --cpu_bind=cores. In addition, I don't know what is the 'aps' argument


srun: error: Unable to create step for job 916208: More processors requested than permitted

I remember the product of -n -c has to be 256.  You can try srun -n 64 -c 4 numactl -m 1 ./ex_modify ...


I?m following the advice as given at slide 33 of https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf

For further info, I?m using LCRC at ANL.

Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/55b97d1d/attachment.html>

From jczhang at mcs.anl.gov  Wed Feb 27 20:24:13 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 28 Feb 2019 02:24:13 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
	<DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CA+MQGp-GKw375HN5rW=usdAi-=aBs==CR-v3og8A9hhrL5zmHg@mail.gmail.com>

Could you provide a compilable and runnable test so I can try it?
--Junchao Zhang


On Wed, Feb 27, 2019 at 7:34 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Thanks, I fixed that, but I?m not actually calling the testScatters() function in my implementation (in the constructor, the only functions I called are setFields and setScatters). So the problem couldn?t have been that?

Best,
Yuyun

From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 10:50 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/b24d085a/attachment-0001.html>

From yyang85 at stanford.edu  Wed Feb 27 21:03:55 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 28 Feb 2019 03:03:55 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <CA+MQGp-GKw375HN5rW=usdAi-=aBs==CR-v3og8A9hhrL5zmHg@mail.gmail.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
	<DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<CA+MQGp-GKw375HN5rW=usdAi-=aBs==CR-v3og8A9hhrL5zmHg@mail.gmail.com>
Message-ID: <DM6PR02MB524254F29066646DFFC882DBD2750@DM6PR02MB5242.namprd02.prod.outlook.com>

Hi Junchao,

This code actually involves a lot of classes and is pretty big. Might be an overkill for me to send everything to you. I'd like to know if I see this sort of error message, which points to this domain file, is it possible that the problem happens in another file (whose operations are linked to this one)? If so, I'll debug a little more and maybe send you more useful information later.

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Zhang, Junchao <jczhang at mcs.anl.gov>
Sent: Wednesday, February 27, 2019 6:24:13 PM
To: Yuyun Yang
Cc: Matthew Knepley; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

Could you provide a compilable and runnable test so I can try it?
--Junchao Zhang


On Wed, Feb 27, 2019 at 7:34 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Thanks, I fixed that, but I?m not actually calling the testScatters() function in my implementation (in the constructor, the only functions I called are setFields and setScatters). So the problem couldn?t have been that?

Best,
Yuyun

From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 10:50 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/364c40dd/attachment.html>

From jczhang at mcs.anl.gov  Wed Feb 27 22:27:16 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 28 Feb 2019 04:27:16 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <DM6PR02MB524254F29066646DFFC882DBD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
	<DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp-GKw375HN5rW=usdAi-=aBs==CR-v3og8A9hhrL5zmHg@mail.gmail.com>
	<DM6PR02MB524254F29066646DFFC882DBD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
Message-ID: <CA+MQGp9kpEHneXq1fXC6vicbUvEu=o1LvK9Ady2NPxR96yAZjw@mail.gmail.com>

Try the following to see if you can catch the bug easily: 1) Get error code for each petsc function and check it with CHKERRQ; 2) Link your code with a petsc library with debugging enabled (configured with --with-debugging=1); 3) Run your code with valgrind

--Junchao Zhang


On Wed, Feb 27, 2019 at 9:04 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Hi Junchao,

This code actually involves a lot of classes and is pretty big. Might be an overkill for me to send everything to you. I'd like to know if I see this sort of error message, which points to this domain file, is it possible that the problem happens in another file (whose operations are linked to this one)? If so, I'll debug a little more and maybe send you more useful information later.

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 6:24:13 PM
To: Yuyun Yang
Cc: Matthew Knepley; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

Could you provide a compilable and runnable test so I can try it?
--Junchao Zhang


On Wed, Feb 27, 2019 at 7:34 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Thanks, I fixed that, but I?m not actually calling the testScatters() function in my implementation (in the constructor, the only functions I called are setFields and setScatters). So the problem couldn?t have been that?

Best,
Yuyun

From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 10:50 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/d1539015/attachment-0001.html>

From yyang85 at stanford.edu  Thu Feb 28 00:02:21 2019
From: yyang85 at stanford.edu (Yuyun Yang)
Date: Thu, 28 Feb 2019 06:02:21 +0000
Subject: [petsc-users] AddressSanitizer: attempting free on address
 which was not malloc()-ed
In-Reply-To: <CA+MQGp9kpEHneXq1fXC6vicbUvEu=o1LvK9Ady2NPxR96yAZjw@mail.gmail.com>
References: <DM6PR02MB52427EA9EE478B5CDCFCA952D2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CAMYG4G=vyvJKbms4ojvVi3TKM4fTNErXLco-iBYqYKyUmhMSbA@mail.gmail.com>
	<DM6PR02MB5242FD9640EFDA4AF266724DD2740@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp8Ay2+et4svo4MVDwv=BkDsmVJHB2d5uFJfcr3zRUMVeA@mail.gmail.com>
	<DM6PR02MB52423B907A23E9504BE53D5AD2750@DM6PR02MB5242.namprd02.prod.outlook.com>
	<CA+MQGp-GKw375HN5rW=usdAi-=aBs==CR-v3og8A9hhrL5zmHg@mail.gmail.com>
	<DM6PR02MB524254F29066646DFFC882DBD2750@DM6PR02MB5242.namprd02.prod.outlook.com>,
	<CA+MQGp9kpEHneXq1fXC6vicbUvEu=o1LvK9Ady2NPxR96yAZjw@mail.gmail.com>
Message-ID: <DM6PR02MB5242F1F0CD6C16409B5389B9D2750@DM6PR02MB5242.namprd02.prod.outlook.com>

Ok, will do that. Thanks a lot!

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Zhang, Junchao <jczhang at mcs.anl.gov>
Sent: Wednesday, February 27, 2019 8:27:17 PM
To: Yuyun Yang
Cc: Matthew Knepley; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

Try the following to see if you can catch the bug easily: 1) Get error code for each petsc function and check it with CHKERRQ; 2) Link your code with a petsc library with debugging enabled (configured with --with-debugging=1); 3) Run your code with valgrind

--Junchao Zhang


On Wed, Feb 27, 2019 at 9:04 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Hi Junchao,

This code actually involves a lot of classes and is pretty big. Might be an overkill for me to send everything to you. I'd like to know if I see this sort of error message, which points to this domain file, is it possible that the problem happens in another file (whose operations are linked to this one)? If so, I'll debug a little more and maybe send you more useful information later.

Best regards,
Yuyun

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 6:24:13 PM
To: Yuyun Yang
Cc: Matthew Knepley; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

Could you provide a compilable and runnable test so I can try it?
--Junchao Zhang


On Wed, Feb 27, 2019 at 7:34 PM Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>> wrote:
Thanks, I fixed that, but I?m not actually calling the testScatters() function in my implementation (in the constructor, the only functions I called are setFields and setScatters). So the problem couldn?t have been that?

Best,
Yuyun

From: Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>>
Sent: Wednesday, February 27, 2019 10:50 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed


On Wed, Feb 27, 2019 at 10:41 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I called VecDestroy() in the destructor for this object ? is that not the right way to do it?
In Domain::testScatters(), you have many VecDuplicate(,&out), You need to VecDestroy(&out) before doing new VecDuplicate(,&out);
How do I implement CHECK ALL RETURN CODES?
For each PETSc function, do ierr = ...;  CHKERRQ(ierr);

From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Wednesday, February 27, 2019 7:24 AM
To: Yuyun Yang <yyang85 at stanford.edu<mailto:yyang85 at stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] AddressSanitizer: attempting free on address which was not malloc()-ed

You call VecDuplicate() a bunch, but VecDestroy() only once in the bottom function. This is wrong.
Also, CHECK ALL RETURN CODES. This is the fastest way to find errors.

   Matt

On Wed, Feb 27, 2019 at 2:06 AM Yuyun Yang via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello team,

I ran into the address sanitizer error that I hope you could help me with. I don?t really know what?s wrong with the way the code frees memory. The relevant code file is attached. The line number following domain.cpp specifically referenced to the vector _q, which seems a little odd, since some other vectors are constructed and freed the same way.

==1719==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x61f0000076c0 in thread T0
    #0 0x7fbf195282ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x7fbf1706f895 in PetscFreeAlign /home/yyy910805/petsc/src/sys/memory/mal.c:87
    #2 0x7fbf1731a898 in VecDestroy_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:788
    #3 0x7fbf1735f795 in VecDestroy /home/yyy910805/petsc/src/vec/vec/interface/vector.c:408
    #4 0x40dd0a in Domain::~Domain() /home/yyy910805/scycle/source/domain.cpp:132
    #5 0x40b479 in main /home/yyy910805/scycle/source/main.cpp:242
    #6 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #7 0x4075d8 in _start (/home/yyy910805/scycle/source/main+0x4075d8)

0x61f0000076c0 is located 1600 bytes inside of 3220-byte region [0x61f000007080,0x61f000007d14)
allocated by thread T0 here:
    #0 0x7fbf19528b32 in __interceptor_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98b32)
    #1 0x7fbf1706f7e0 in PetscMallocAlign /home/yyy910805/petsc/src/sys/memory/mal.c:41
    #2 0x7fbf17073022 in PetscTrMallocDefault /home/yyy910805/petsc/src/sys/memory/mtr.c:183
    #3 0x7fbf170710a1 in PetscMallocA /home/yyy910805/petsc/src/sys/memory/mal.c:397
    #4 0x7fbf17326fb0 in VecCreate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec3.c:35
    #5 0x7fbf1736f560 in VecSetType /home/yyy910805/petsc/src/vec/vec/interface/vecreg.c:51
    #6 0x7fbf1731afae in VecDuplicate_Seq /home/yyy910805/petsc/src/vec/vec/impls/seq/bvec2.c:807
    #7 0x7fbf1735eff7 in VecDuplicate /home/yyy910805/petsc/src/vec/vec/interface/vector.c:379
    #8 0x4130de in Domain::setFields() /home/yyy910805/scycle/source/domain.cpp:431
    #9 0x40c60a in Domain::Domain(char const*) /home/yyy910805/scycle/source/domain.cpp:57
    #10 0x40b433 in main /home/yyy910805/scycle/source/main.cpp:242
    #11 0x7fbf14d2082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: bad-free ??:0 __interceptor_free
==1719==ABORTING

Thanks very much!
Yuyun


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/eb6f44f0/attachment.html>

From hongzhang at anl.gov  Thu Feb 28 00:50:54 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Thu, 28 Feb 2019 06:50:54 +0000
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
	<CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
	<CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
Message-ID: <27578CF7-FF30-4461-9B30-3BE5B41585C8@anl.gov>

For LCRC, it is better to follow the example on their website (http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/)

#!/bin/bash

#SBATCH -J mympijobname
#SBATCH -A myaccount
#SBATCH -p knlall
#SBATCH -C knl,quad,cache
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH -t 00:15:00

export I_MPI_FABRICS=shm:tmi
srun ./ex_modify

I would not recommend you to use flat mode unless you really want to seriously optimize your code for best performance on KNL.

When configuring petsc on KNL, adding options --with-avx512-kernels --with-memalign=64 may speed up your code a bit.

Hong (Mr.)

On Feb 27, 2019, at 5:02 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


Hi Junchao,

I?m confused with the syntax. If I submit the following as my job script, I get an error :

#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00

module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify -ts_type cn -prop_steps 25 -pc_type gamg -ts_monitor -log_view


The error is :
srun: cluster configuration lacks support for cpu binding
srun: error: Unable to create step for job 916208: More processors requested than permitted

I?m following the advice as given at slide 33 of https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf

For further info, I?m using LCRC at ANL.

Thank You,
Sajid Ali
Applied Physics
Northwestern University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/3f3f62fb/attachment-0001.html>

From eda.oktay at metu.edu.tr  Thu Feb 28 02:14:43 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Thu, 28 Feb 2019 11:14:43 +0300
Subject: [petsc-users] Error in using MatTranspose
Message-ID: <CA+vu_Xjd_CaXg=qfJ40yWivGG-39Xb_oEm5U6CKZzQRV7xhuKQ@mail.gmail.com>

Hello,

I am trying to use MatTranspose function as in below (last row) but I get
an error like this:
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./FINAL on a arch-linux2-c-debug named
localhost.localdomain by edaoktay Wed Feb 27 16:36:34 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
--download-metis --download-parmetis --download-superlu_dist
--download-slepc --download-mpich
[0]PETSC ERROR: #1 MatTranspose() line 4821 in
/home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 main() line 185 in
/home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/FINAL.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -f
/home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/small
[0]PETSC ERROR: -unweighted
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------

I understood that error states that there is an unassembled matrix but
there is not, as it can be seen below, I defined NSymmA and I want to get
NSymmAtr.

This not the first time I use MatTranspose but I only get this error here.
What can be the reason?

Best regards,

Eda


ierr = MatGetOwnershipRange(L,&mm,&nn);CHKERRQ(ierr);


        ierr =
MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&NSymmA);CHKERRQ(ierr);
        for(rw = mm; rw<nn; ++rw){

            ierr = MatGetRow(L,rw,&ncols,&cols,&vals);CHKERRQ(ierr);

            s = sizeof(vals);

            ierr = PetscMalloc1(s,&arr);CHKERRQ(ierr);

            for(j=0;j<s;++j){

                arr[j]=-1;
            }
            ierr =
MatSetValues(NSymmA,1,&rw,ncols,cols,arr,INSERT_VALUES);CHKERRQ(ierr);
            ierr = MatRestoreRow(L,rw,&ncols,&cols,&vals);CHKERRQ(ierr);
        }

        ierr =
MatTranspose(NSymmA,MAT_INITIAL_MATRIX,&NSymmAtr);CHKERRQ(ierr);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/74500a58/attachment.html>

From eda.oktay at metu.edu.tr  Thu Feb 28 02:21:47 2019
From: eda.oktay at metu.edu.tr (Eda Oktay)
Date: Thu, 28 Feb 2019 11:21:47 +0300
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <9335798C-7B6C-4B94-983C-D9F6EBF125D5@dsic.upv.es>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<9335798C-7B6C-4B94-983C-D9F6EBF125D5@dsic.upv.es>
Message-ID: <CA+vu_XjjzmL0J3FF8HWO4+=6WbtLLHzrLWOGqvWY7yawKpm5Ew@mail.gmail.com>

Dear Professor Roman,

Thank you for your answer. I used PetscBinaryWrite.m file and converted
matrices into binary format but I have a question:

When I look at Properties of the binary files Petsc have, their type is
stated as "Program (application/octet-stream)", however when I checked the
binary files I created their type is stated as "Binary
(application/octet-stream)". Can this cause a problem, why they are
different?

Eda

Jose E. Roman <jroman at dsic.upv.es>, 12 ?ub 2019 Sal, 12:05 tarihinde ?unu
yazd?:

> It is better to convert the matrices to PETSc binary format first. One
> easy way is to read them into Matlab with mmread.m and write with PETSc's
> PetscBinaryWrite.m. This can be done similarly in python.
> Jose
>
>
> > El 12 feb 2019, a las 8:50, Eda Oktay via petsc-users <
> petsc-users at mcs.anl.gov> escribi?:
> >
> > Hello,
> >
> > I am trying to load matrix in Matrix Market format. I found an example
> on mat file  (ex78) whih can be tested by using .dat file. Since .dat file
> and .mtx file are similar  in structure (specially afiro_A.dat file is
> similar to amesos2_test_mat0.mtx since they both have 3 columns and the
> columns represent the same properties), I tried to run ex78 by using
> amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error
> "Badly formatted input file". Here is the full error message:
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Badly formatted input file
> >
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named
> 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [0]PETSC ERROR: #1 main() line 73 in
> /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
> > [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -Ain
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
> > :
> > system msg for write_line failure : Bad file descriptor
> >
> > I know there is also an example (ex72) for Matrix Market format but in
> description, it is only proper for symmmetric and lower triangle, so I
> decided to use ex78.
> >
> > Best regards,
> >
> > Eda
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/bc818fb0/attachment.html>

From jroman at dsic.upv.es  Thu Feb 28 02:38:46 2019
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 28 Feb 2019 09:38:46 +0100
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+vu_XjjzmL0J3FF8HWO4+=6WbtLLHzrLWOGqvWY7yawKpm5Ew@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
	<9335798C-7B6C-4B94-983C-D9F6EBF125D5@dsic.upv.es>
	<CA+vu_XjjzmL0J3FF8HWO4+=6WbtLLHzrLWOGqvWY7yawKpm5Ew@mail.gmail.com>
Message-ID: <74A1D72B-97ED-48F2-8602-1F10E1A09347@dsic.upv.es>

Don't know why it should cause problems. I'd rather try before asking.

> El 28 feb 2019, a las 9:21, Eda Oktay <eda.oktay at metu.edu.tr> escribi?:
> 
> Dear Professor Roman,
> 
> Thank you for your answer. I used PetscBinaryWrite.m file and converted matrices into binary format but I have a question:
> 
> When I look at Properties of the binary files Petsc have, their type is stated as "Program (application/octet-stream)", however when I checked the binary files I created their type is stated as "Binary (application/octet-stream)". Can this cause a problem, why they are different?
> 
> Eda
> 
> Jose E. Roman <jroman at dsic.upv.es>, 12 ?ub 2019 Sal, 12:05 tarihinde ?unu yazd?:
> It is better to convert the matrices to PETSc binary format first. One easy way is to read them into Matlab with mmread.m and write with PETSc's PetscBinaryWrite.m. This can be done similarly in python.
> Jose
> 
> 
> > El 12 feb 2019, a las 8:50, Eda Oktay via petsc-users <petsc-users at mcs.anl.gov> escribi?:
> > 
> > Hello,
> > 
> > I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:
> > 
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [0]PETSC ERROR: Badly formatted input file
> > 
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> > [0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr by edaoktay Tue Feb 12 10:47:58 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
> > [0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
> > [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
> > :
> > system msg for write_line failure : Bad file descriptor
> > 
> > I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.
> > 
> > Best regards,
> > 
> > Eda 
> 


From t.appel17 at imperial.ac.uk  Thu Feb 28 05:54:55 2019
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Thu, 28 Feb 2019 11:54:55 +0000
Subject: [petsc-users] About DMDA (and extracting its ordering)
In-Reply-To: <CAMYG4GkHUnRhbYLTwptwqmtjDkTqQfquJgby_+FSSKoCTHMJLQ@mail.gmail.com>
References: <b0f81fd5-080b-726c-c4e1-268e7ab22978@imperial.ac.uk>
	<CAMYG4GnUKkjLm4s-WrexvmtOW0h7GHzH4=qnruWdad-6pnFrdg@mail.gmail.com>
	<4f02c398-acd6-7e97-d772-e335a6accc8d@imperial.ac.uk>
	<CAMYG4GnAVuXkcXuS32o4NCzER7Mr+teXia-=B8aM=qUxTCJJ3Q@mail.gmail.com>
	<A4853CA7-008D-4036-97B6-5B4A5BFCE0EC@ic.ac.uk>
	<CAMYG4Gm3-RjLD_9aXhJNKbRWJzjnSz9eH7eP2wUXs+u-OEgMVg@mail.gmail.com>
	<069608FD-538E-4C15-AA2D-09CE6E4BD94D@ic.ac.uk>
	<CAMYG4G=Km94AmjTpL8bX0rNL4DvTvWFtVvA__-yUK1LPQJKqvQ@mail.gmail.com>
	<7ABDDB41-0417-42D0-8BF6-A7F74C7285FD@imperial.ac.uk>
	<CAMYG4GkHUnRhbYLTwptwqmtjDkTqQfquJgby_+FSSKoCTHMJLQ@mail.gmail.com>
Message-ID: <B314E8C6-487F-40DD-8085-191FF6F89545@ic.ac.uk>

Hi Matthew,

I?ve just attached on that page https://bitbucket.org/petsc/petsc/issues/262 the same source code written in C that runs just fine from what I can see.

Which means there?s either a bug in ISLocalToGlobalMappingGetIndicesF90 or something missing in the documentation.

Best,

Thibaut


On 25 Feb 2019, at 12:03, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Mon, Feb 25, 2019 at 3:19 AM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Hi Matthew,

Yes I need F90 and the syntax in the file / yours
PetscInt, pointer :: id_ltog(:)

Is the exact same as
PetscInt, dimension(:), pointer :: id_ltog

They?re both modern fortran ?correct?

Anyways I tried both and I still get the same error message.

Okay, that file I sent the link to is tested every night, so I suspect a compiler bug in your version.
Please send your entire example so I can run it here. Also, let us know what Fortran compiler
and version you are using again.

  Thanks,

     Matt

Thibaut

On 24 Feb 2019, at 23:38, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Is that for the id_ltog argument? I tried declaring it as IS, and you can?t declare an deferred-shape array as target. Same message. XXX(:) is syntactic sugar for dimension(:) :: XXX

I?m really confused because this
https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html
doesn?t use the xxxF90 variants of the routines we?re talking about?

You can use the F77 stuff if you want. I thought you needed F90.  For F90, I think you need to declare it

  PetscInt, pointer :: id_ltog(:)

like it is in the link I sent you.

  Thanks,

     Matt

Thibaut

On 24 Feb 2019, at 22:50, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <t.appel17 at imperial.ac.uk<mailto:t.appel17 at imperial.ac.uk>> wrote:
Hi Matthew,

With the following data declaration and routines calls:

I am not a Fortran90 expert, but your declaration looks different than the ones I have seen work:

  https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90

  Thanks,

     Matt


  DM :: da
  ISLocalToGlobalMapping :: ltog
  PetscInt, DIMENSION(:), POINTER :: id_ltog

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)


I get, with the most recent Intel Fortran compiler and a fresh ?git pull? PETsc:

forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is not associated with a target


Thibaut

On 22 Feb 2019, at 15:13, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I reckon that what corresponds best to my situation is to create my own MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself and fill my values with MatSetValues.

However, what is the correct way to call ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something missing in the documentation as my code just won't work.

Some examples include a PetscOffset argument which is not present in the documentation and there is no manual page for ISLocalToGlobalMappingGetIndicesF90: the only information I could find is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to allocate the index array yourself? I tried declaring it as a pointer too. I keep getting an error

1) There was a bug in that file, but I don't think it should have affected you. I will put in a PR

2)  You are supposed to pass in a pointer, unallocated. We manage all the allocation, which is why you call Restore. What happens when you do that?

  Thanks,

     Matt
[0]PETSC ERROR: #1 User provided function() line 0 in User file
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0

Minimal example:

PROGRAM test_dmda

#include <petsc/finclude/petscdm.h>
#include <petsc/finclude/petscdmda.h>

  USE PetscDMDA

  IMPLICIT NONE

  PetscErrorCode :: ierr
  PetscInt       :: irank, icx, icy, lx, ly, n_ltog
  DM             :: da
  ISLocalToGlobalMapping :: ltog

  PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog
  !PetscInt, DIMENSION(:), POINTER :: id_ltog

  PetscInt, PARAMETER :: nx=24, ny=9, neq=4
  CHARACTER(LEN=100) :: err_msg

  CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)

  CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)

  CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &
                    PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
  CHKERRA(ierr)

  CALL DMSetUp(da,ierr)
  CHKERRA(ierr)

  CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)
  CHKERRA(ierr)

  WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left corner is icx=', icx, &
                          'and icy=', icy, ' - the width is lx=', lx, ' and ly=', ly

  CALL DMGetLocalToGlobalMapping(da,ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)
  CHKERRA(ierr)

  ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)

  CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)
  CHKERRA(ierr)

  DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)

  CALL DMDestroy(da,ierr)
  CHKERRA(ierr)

  CALL PetscFinalize(ierr)

END PROGRAM test_dmda



On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Dear PETSc developers/users,

I?m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I?ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.

However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory ?optimally?. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about it

?These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization?

And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?

What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop on that in my preallocation and assembly routines.

You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.

  Thanks,

     Matt

Thanks very much,

Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/95257278/attachment-0001.html>

From sajidsyed2021 at u.northwestern.edu  Thu Feb 28 08:10:55 2019
From: sajidsyed2021 at u.northwestern.edu (Sajid Ali)
Date: Thu, 28 Feb 2019 08:10:55 -0600
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <27578CF7-FF30-4461-9B30-3BE5B41585C8@anl.gov>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
	<CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
	<CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
	<27578CF7-FF30-4461-9B30-3BE5B41585C8@anl.gov>
Message-ID: <CAOGsD9gvvKUXy0aziJB-soPVUVBpcC1LUgZPeyy_y9wWg1Jn1w@mail.gmail.com>

Hi Hong,

Thanks for the advice. I see that the example takes ~180 seconds to run but
I can't see the DRAM vs MCDRAM info from Intel APS. I'll try to fix the
profiling and get back with further questions.

Also, the intel-mpi manpages say that the use of tmi is now deprecated :
https://software.intel.com/en-us/mpi-developer-guide-linux-fabrics-control


Thank You,
Sajid Ali
Applied Physics
Northwestern University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/051fedd5/attachment.html>

From knepley at gmail.com  Thu Feb 28 08:34:48 2019
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 28 Feb 2019 09:34:48 -0500
Subject: [petsc-users] Error in using MatTranspose
In-Reply-To: <CA+vu_Xjd_CaXg=qfJ40yWivGG-39Xb_oEm5U6CKZzQRV7xhuKQ@mail.gmail.com>
References: <CA+vu_Xjd_CaXg=qfJ40yWivGG-39Xb_oEm5U6CKZzQRV7xhuKQ@mail.gmail.com>
Message-ID: <CAMYG4Gkv=1Nj5155-Ywb3QN58M1OO32npJDzFo_2pGOjuDbOSg@mail.gmail.com>

On Thu, Feb 28, 2019 at 3:15 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello,
>
> I am trying to use MatTranspose function as in below (last row) but I get
> an error like this:
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled matrix
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./FINAL on a arch-linux2-c-debug named
> localhost.localdomain by edaoktay Wed Feb 27 16:36:34 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [0]PETSC ERROR: #1 MatTranspose() line 4821 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 main() line 185 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/FINAL.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/small
> [0]PETSC ERROR: -unweighted
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
>
> I understood that error states that there is an unassembled matrix but
> there is not, as it can be seen below, I defined NSymmA and I want to get
> NSymmAtr.
>
> This not the first time I use MatTranspose but I only get this error here.
> What can be the reason?
>
> Best regards,
>
> Eda
>
>
> ierr = MatGetOwnershipRange(L,&mm,&nn);CHKERRQ(ierr);
>
>
>         ierr =
> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&NSymmA);CHKERRQ(ierr);
>         for(rw = mm; rw<nn; ++rw){
>
>             ierr = MatGetRow(L,rw,&ncols,&cols,&vals);CHKERRQ(ierr);
>
>             s = sizeof(vals);
>
>             ierr = PetscMalloc1(s,&arr);CHKERRQ(ierr);
>
>             for(j=0;j<s;++j){
>
>                 arr[j]=-1;
>             }
>             ierr =
> MatSetValues(NSymmA,1,&rw,ncols,cols,arr,INSERT_VALUES);CHKERRQ(ierr);
>             ierr = MatRestoreRow(L,rw,&ncols,&cols,&vals);CHKERRQ(ierr);
>         }
>

 You have to assemble here since you called MatSetValues().

  Matt

        ierr =
> MatTranspose(NSymmA,MAT_INITIAL_MATRIX,&NSymmAtr);CHKERRQ(ierr);
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/8571cf32/attachment.html>

From jczhang at mcs.anl.gov  Thu Feb 28 09:18:30 2019
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Thu, 28 Feb 2019 15:18:30 +0000
Subject: [petsc-users] Problem in loading Matrix Market format
In-Reply-To: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
References: <CA+vu_XjN42jvfnnR5AS=Pb=EOQSMmMsyKxQW1GbbZhm0hMwo0w@mail.gmail.com>
Message-ID: <CA+MQGp9D2wtDbHHBRpHeAT5iF_OoVv6V2YrNpK008inLSP-E8w@mail.gmail.com>

Eda,
  An update to ex72 is merged to PETSc master branch just now. It now can read matrices either symmetric or non-symmetric in Matrix Market format, and output a petsc binary matrix in MATSBAIJ format (for symmetric) or MATAIJ format (for non-symmetric). See help in source code for usage.
--Junchao Zhang


On Tue, Feb 12, 2019 at 1:50 AM Eda Oktay via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello,

I am trying to load matrix in Matrix Market format. I found an example on mat file  (ex78) whih can be tested by using .dat file. Since .dat file and .mtx file are similar  in structure (specially afiro_A.dat file is similar to amesos2_test_mat0.mtx since they both have 3 columns and the columns represent the same properties), I tried to run ex78 by using amesos2_test_mat0.mtx instead of afiro_A.dat. However, I got the error "Badly formatted input file". Here is the full error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Badly formatted input file

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
[0]PETSC ERROR: ./ex78 on a arch-linux2-c-debug named 7330.wls.metu.edu.tr<http://7330.wls.metu.edu.tr> by edaoktay Tue Feb 12 10:47:58 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich
[0]PETSC ERROR: #1 main() line 73 in /home/edaoktay/petsc-3.10.3/src/mat/examples/tests/ex78.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -Ain /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/amesos2_test_mat0.mtx
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

I know there is also an example (ex72) for Matrix Market format but in description, it is only proper for symmmetric and lower triangle, so I decided to use ex78.

Best regards,

Eda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/52213f27/attachment.html>

From hongzhang at anl.gov  Thu Feb 28 10:05:40 2019
From: hongzhang at anl.gov (Zhang, Hong)
Date: Thu, 28 Feb 2019 16:05:40 +0000
Subject: [petsc-users] Direct PETSc to use MCDRAM on KNL and other
 optimizations for KNL
In-Reply-To: <CAOGsD9gvvKUXy0aziJB-soPVUVBpcC1LUgZPeyy_y9wWg1Jn1w@mail.gmail.com>
References: <CAOGsD9gGREKnp-kSG54-W7E60Ln+4i_O1DLGz12QBcnjHOjf4A@mail.gmail.com>
	<CA+MQGp9vGq6zCawBpDLEba=YE5L+1dXkaodqM54nkLXW9Ov4DQ@mail.gmail.com>
	<CAOGsD9h6jHd_TnOBt=WiacWnLhn1OLyMgd7aK_PxZxAVdiL8=A@mail.gmail.com>
	<27578CF7-FF30-4461-9B30-3BE5B41585C8@anl.gov>
	<CAOGsD9gvvKUXy0aziJB-soPVUVBpcC1LUgZPeyy_y9wWg1Jn1w@mail.gmail.com>
Message-ID: <CF9DEE4E-74C9-40B2-8DA4-7058E034ED03@anl.gov>



On Feb 28, 2019, at 6:10 AM, Sajid Ali <sajidsyed2021 at u.northwestern.edu<mailto:sajidsyed2021 at u.northwestern.edu>> wrote:

Hi Hong,

Thanks for the advice. I see that the example takes ~180 seconds to run but I can't see the DRAM vs MCDRAM info from Intel APS. I'll try to fix the profiling and get back with further questions.

MCDRAM has 4x higher bandwidth than DRAM, so the improvement you see from your example looks very reasonable. Note that in cache mode MCDRAM acts as L3 cache while in flat mode it is used as another level of memory.

Hong (Mr.)



Also, the intel-mpi manpages say that the use of tmi is now deprecated : https://software.intel.com/en-us/mpi-developer-guide-linux-fabrics-control


Thank You,
Sajid Ali
Applied Physics
Northwestern University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/df54a118/attachment-0001.html>

From k_burkart at yahoo.com  Thu Feb 28 12:55:31 2019
From: k_burkart at yahoo.com (Klaus Burkart)
Date: Thu, 28 Feb 2019 18:55:31 +0000 (UTC)
Subject: [petsc-users] What's the correct syntax for
 PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true"); ?
References: <128225664.7096832.1551380131487.ref@mail.yahoo.com>
Message-ID: <128225664.7096832.1551380131487@mail.yahoo.com>

Hello,
I try to use the Hypre boomeramg preconditioner but I keep getting type conversion errors like this one for PetscOptionsSetValue(...):
Fehler: ?const char*? kann nicht nach ?PetscOptions? {aka ?_n_PetscOptions*?} umgewandelt werden
???? PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true");

This is my setup:

??? PCSetType(pc,PCHYPRE);
??? PCHYPRESetType(pc,"boomeramg");
??? PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true");

What's the correct syntax for PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true"); ?

Klaus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/717b1811/attachment.html>

From balay at mcs.anl.gov  Thu Feb 28 13:02:57 2019
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 28 Feb 2019 19:02:57 +0000
Subject: [petsc-users] What's the correct syntax for
 PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true"); ?
In-Reply-To: <128225664.7096832.1551380131487@mail.yahoo.com>
References: <128225664.7096832.1551380131487.ref@mail.yahoo.com>
	<128225664.7096832.1551380131487@mail.yahoo.com>
Message-ID: <alpine.LFD.2.21.1902281300590.14161@sb>

On Thu, 28 Feb 2019, Klaus Burkart via petsc-users wrote:

> Hello,
> I try to use the Hypre boomeramg preconditioner but I keep getting type conversion errors like this one for PetscOptionsSetValue(...):
> Fehler: ?const char*? kann nicht nach ?PetscOptions? {aka ?_n_PetscOptions*?} umgewandelt werden
> ???? PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true");
> 
> This is my setup:
> 
> ??? PCSetType(pc,PCHYPRE);
> ??? PCHYPRESetType(pc,"boomeramg");
> ??? PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true");
> 
> What's the correct syntax for PetscOptionsSetValue("-pc_hypre_boomeramg_no_CF","true"); ?

Check examples at: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsSetValue.html

  PetscOptionsSetValue(NULL,"-pc_hypre_boomeramg_no_CF","true");

Satish

From danyang.su at gmail.com  Thu Feb 28 14:33:11 2019
From: danyang.su at gmail.com (Danyang Su)
Date: Thu, 28 Feb 2019 12:33:11 -0800
Subject: [petsc-users] Fortran example for PETSc global to natural
 ordering test
In-Reply-To: <CAMYG4GkGUBE6P0pnd6LOF28zzuPtDFcRKYZ-xm_Ec_x3g+AdUQ@mail.gmail.com>
References: <7d7a300e-dc7f-b2f1-2cd1-4291b520116c@gmail.com>
	<CAMYG4Gnpjvx6VXBy43drpHRJY1WQeYwVy+znnmmK5-mA58uSUg@mail.gmail.com>
	<d398ac62-16a0-1551-b776-604cbda5c345@gmail.com>
	<CAMYG4GmRXGMm7fP+aCfEpQiD_tX-fHpy2RLBeukFBk3O8zruGQ@mail.gmail.com>
	<0cc7da60-3450-8642-b21d-5eb37890ed47@gmail.com>
	<CAMYG4GkZy2dfP-0LHYxwK0_+Kw7qGu_VaFbqo1MbqJ6hwRQAqw@mail.gmail.com>
	<a1d6d99b-9329-be15-9c3e-c2d71416af92@gmail.com>
	<1b61895c-e65c-91f2-10aa-d7e8b653c1b9@gmail.com>
	<CAMYG4GkGUBE6P0pnd6LOF28zzuPtDFcRKYZ-xm_Ec_x3g+AdUQ@mail.gmail.com>
Message-ID: <672fdecc-a30d-90d2-c5d0-7f55a676990e@gmail.com>

Hi Matt,

I have further questions regarding global to natural ordering using the 
latest PETSc-dev version.

1) Is the mesh reordered when DMPlexCreateFromCellList is used? I made 
some tests on this but found the natural order was not same as the mesh 
I fed DMPlex. Probably I made something wrong using 
DMPlexGlobalToNaturalBegin(End)?

2) If I pass null labels to DMPlexCreateSection, the get label cannot 
work and the returned value is -1, which is as expected since no labels 
are passed. However, if I pass valid labels to DMPlexCreateSection, I 
get vector size 1 for all the processors when use DMCreateGlobalVector. 
When I use PETSc-3.10.3 or earlier version, labels are set separately 
and it works fine.

Attached is the example I use, an updated version of fortran example to 
read mesh from external file, distribute over processors and then check 
global to natural ordering.

To compile the code, type 'make natural'. Below is the screen output 
when run the code using 4 processors. The local node index (node-loc) 
and natural node index (node-nat) look fine, but the coordinates do not 
match, unfortunately.

+rank??? 0 node-loc??? 1 node-petsc??? 0 node-nat?? 14 node-label?? -1? 
coord-loc? 0.000E+00? 0.000E+00? 0.200E+01 coord-nat? 0.100E+01? 
0.000E+00? 0.000E+00
+rank??? 0 node-loc??? 2 node-petsc??? 1 node-nat?? 30 node-label?? -1? 
coord-loc? 0.800E+00? 0.000E+00? 0.120E+01 coord-nat? 0.110E+01? 
0.000E+00? 0.800E+00
+rank??? 0 node-loc??? 3 node-petsc??? 2 node-nat?? 31 node-label?? -1? 
coord-loc? 0.000E+00? 0.000E+00? 0.150E+01 coord-nat? 0.367E+00? 
0.000E+00? 0.123E+01
+rank??? 1 node-loc??? 1 node-petsc?? 13 node-nat?? 47 node-label?? -1? 
coord-loc? 0.000E+00? 0.000E+00? 0.000E+00 coord-nat? 0.197E+01? 
0.000E+00? 0.467E+00
+rank??? 1 node-loc??? 2 node-petsc?? 14 node-nat?? 48 node-label?? -1? 
coord-loc? 0.200E+00? 0.000E+00? 0.800E+00 coord-nat? 0.632E+00? 
0.000E+00? 0.949E+00

Is there any chance for you to make a test or point out any mistakes 
I've made?

Thanks,

Danyang

On 2018-12-04 11:06 a.m., Matthew Knepley wrote:
> On Mon, Dec 3, 2018 at 8:32 PM Danyang Su <danyang.su at gmail.com 
> <mailto:danyang.su at gmail.com>> wrote:
>
>     Hi Matt,
>
>     Attached is the test example with source code, makefile, data and
>     screen output I wrote this afternoon. This example reads 2D mesh
>     from vtk file and then distribute over all processors. I can get
>     correct global order of local nodes after distribution, but the
>     natural order of local nodes is always zero after using
>     DMPlexGlobalToNaturalBegin/End(). Sounds like it does not take
>     effect.
>
>     Would you mind helping me to check/test this code at you most
>     convenience?
>
> On 2 procs, I get an SEGV
>
> rank1 local nodes with ghost 47 local cells with ghost 75
>
> rank0 local nodes with ghost 52 local cells with ghost 78
>
> [1]PETSC ERROR: 
> ------------------------------------------------------------------------
>
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
> probably memory access out of range
>
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [1]PETSC ERROR: or see 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac 
> OS X to find memory corruption errors
>
> [1]PETSC ERROR: likely location of problem given in stack below
>
> [1]PETSC ERROR: ---------------------Stack Frames 
> ------------------------------------
>
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not 
> available,
>
> [1]PETSC ERROR: INSTEAD the line number of the start of the function
>
> [1]PETSC ERROR: is given.
>
> [1]PETSC ERROR: --------------------- Error Message 
> --------------------------------------------------------------
>
> [1]PETSC ERROR: Signal received
>
> [1]PETSC ERROR: See 
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>
> [1]PETSC ERROR: Petsc Development GIT revision: 
> v3.9.3-1021-g8625415GIT Date: 2018-08-02 12:57:14 -0500
>
> [1]PETSC ERROR: Unknown Name on a arch-master-debug named 
> MATTHEW-KNEPLEYs-MacBook-Air-2.local by knepley Tue Dec4 14:00:46 2018
>
> [1]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug 
> --download-chaco 
> --download-cmake=/Users/knepley/Downloads/cmake-3.7.2.tar.gz 
> --download-ctetgen --download-eigen --download-fftw --download-hdf5 
> --download-med --download-metis --download-mpich --download-netcdf 
> --download-p4est --download-parmetis --download-pnetcdf 
> --download-superlu_dist --download-triangle 
> --with-cc="/Users/knepley/MacSoftware/bin/ccache gcc 
> -Qunused-arguments" --with-cxx="/Users/knepley/MacSoftware/bin/ccache 
> g++ -Qunused-arguments" 
> --with-fc="/Users/knepley/MacSoftware/bin/ccache gfortran" 
> --with-shared-libraries
>
> [1]PETSC ERROR: #1 User provided function() line 0 inunknown file
>
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - 
> process 1
>
> ------------------------------------------------------------------------
>
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
> probably memory access out of range
>
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac 
> OS X to find memory corruption errors
>
> [0]PETSC ERROR: likely location of problem given in stack below
>
> [0]PETSC ERROR: ---------------------Stack Frames 
> ------------------------------------
>
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not 
> available,
>
> [0]PETSC ERROR: INSTEAD the line number of the start of the function
>
> [0]PETSC ERROR: is given.
>
> [0]PETSC ERROR: --------------------- Error Message 
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Signal received
>
> [0]PETSC ERROR: See 
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>
> [0]PETSC ERROR: Petsc Development GIT revision: 
> v3.9.3-1021-g8625415GIT Date: 2018-08-02 12:57:14 -0500
>
> [0]PETSC ERROR: Unknown Name on a arch-master-debug named 
> MATTHEW-KNEPLEYs-MacBook-Air-2.local by knepley Tue Dec4 14:00:46 2018
>
> [0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug 
> --download-chaco 
> --download-cmake=/Users/knepley/Downloads/cmake-3.7.2.tar.gz 
> --download-ctetgen --download-eigen --download-fftw --download-hdf5 
> --download-med --download-metis --download-mpich --download-netcdf 
> --download-p4est --download-parmetis --download-pnetcdf 
> --download-superlu_dist --download-triangle 
> --with-cc="/Users/knepley/MacSoftware/bin/ccache gcc 
> -Qunused-arguments" --with-cxx="/Users/knepley/MacSoftware/bin/ccache 
> g++ -Qunused-arguments" 
> --with-fc="/Users/knepley/MacSoftware/bin/ccache gfortran" 
> --with-shared-libraries
>
> [0]PETSC ERROR: #1 User provided function() line 0 inunknown file
>
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> which valgrind says comes from DMPlexCreateSection()
>
> rank0 local nodes with ghost 52 local cells with ghost 78
>
> rank1 local nodes with ghost 47 local cells with ghost 75
>
> ==14766== Invalid write of size 4
>
> ==14766==at 0x1151A3B: dmplexcreatesection_ (zplexsectionf90.c:20)
>
> ==14766==by 0x1000042FA: MAIN__ (in ./natural)
>
> ==14766==by 0x10000497D: main (in ./natural)
>
> ==14766==Address 0x600001000 is not stack'd, malloc'd or (recently) free'd
>
> ==14766==
>
> ==14767== Invalid write of size 4
>
> ==14767==at 0x1151A3B: dmplexcreatesection_ (zplexsectionf90.c:20)
>
> ==14767==by 0x1000042FA: MAIN__ (in ./natural)
>
> ==14767==by 0x10000497D: main (in ./natural)
>
> ==14767==Address 0x600001000 is not stack'd, malloc'd or (recently) free'd
>
> ==14767==
>
> I cannot run the debugger on it because command line arguments are not 
> working with
> my Fortran compiler (Ugh). Do you see this error?
>
> ? Thanks,
>
> ? ? Matt
>
>     Thanks,
>
>     Danyang
>
>     On 2018-12-03 1:12 p.m., Danyang Su wrote:
>>
>>
>>     On 2018-12-03 12:56 p.m., Matthew Knepley wrote:
>>>     On Mon, Dec 3, 2018 at 3:40 PM Danyang Su <danyang.su at gmail.com
>>>     <mailto:danyang.su at gmail.com>> wrote:
>>>
>>>
>>>         On 2018-12-03 12:03 p.m., Matthew Knepley wrote:
>>>>         On Mon, Dec 3, 2018 at 2:27 PM Danyang Su
>>>>         <danyang.su at gmail.com <mailto:danyang.su at gmail.com>> wrote:
>>>>
>>>>             Hi Matt,
>>>>
>>>>             Thanks.
>>>>
>>>>             BTW: DmPlexGetVertexNumbering now can work using the
>>>>             latest develop version. But the index is not in natural
>>>>             ordering when DMSetUseNatural is called. That's why I
>>>>             want to use PetscSFDistributeSection to check if I miss
>>>>             anything in the code.
>>>>
>>>>         Can you explain that a little more? Maybe you can just push
>>>>         forward what you want using the migrationSF.
>>>
>>>         Hi Matt,
>>>
>>>         Since I cannot figure what is wrong or missing in my code, I
>>>         followed an old ex26.c example in
>>>         src/dm/impls/plex/examples/tests to create similar code as
>>>         shown below to test global to natural ordering. The code may
>>>         be ugly with unnecessary functions in it. Using
>>>         DmPlexGetVertexNumbering, I can get the value but it is not
>>>         in natural order, instead, it is still in default PETSc
>>>         order without calling DMSetUseNatural(dm,PETSC_TRUE,ierr).
>>>
>>>     I do not understand what you are doing below. You just need to call
>>>
>>>     ? ? ierr = DMSetUseNatural(dm,PETSC_TRUE);CHKERRQ(ierr);
>>>     ? ? ierr = DMPlexDistribute(dm,0,&migrationSF,&pdm);CHKERRQ(ierr);
>>>     ? ? if (pdm) {
>>>     ? ? ? ierr = DMPlexSetMigrationSF(pdm,migrationSF);CHKERRQ(ierr);
>>>     ? ? }
>>>     and the DMGlobalToNaturalBegin/End() should work.
>>
>>     You mean to use DMPlexGlobalToNaturalBegin/End(), right? That's
>>     what I tried at first, but without success.
>>
>>     I will create a test example to make further check if I can
>>     reproduce the problem.
>>
>>     Thanks,
>>
>>     Danyang
>>
>>>
>>>     ? Thanks,
>>>
>>>     ? ? ?Matt
>>>
>>>         ? ? ? ? ? if (rank == 0) then
>>>
>>>         ??????????? call
>>>         DMPlexCreateFromCellList(Petsc_Comm_World,ndim,num_cells,
>>>         num_nodes,num_nodes_per_cell,??? &
>>>         Petsc_False,dmplex_cells,ndim, dmplex_verts,dm,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>         ????????? else
>>>         ??????????? call
>>>         DMPlexCreateFromCellList(Petsc_Comm_World,ndim,0,
>>>         0,num_nodes_per_cell,??? &
>>>         Petsc_False,dmplex_cells,ndim,dmplex_verts,dm,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>         ????????? end if
>>>
>>>         ????????? if (nprocs > 1) then
>>>         ??????????? call DMSetUseNatural(dm,PETSC_TRUE,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>         ????????? end if
>>>
>>>         ????????? call DMPlexDistribute(dm,stencil_width, &
>>>         migrationsf,distributedMesh,ierr)
>>>         ????????? CHKERRQ(ierr)
>>>
>>>         ????????? if (distributedMesh /= PETSC_NULL_DM) then
>>>         ??????????? call
>>>         PetscSFCreateInverseSF(migrationsf,migrationsf_inv,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call
>>>         DMCreateGlobalToNatural(distributedMesh,migrationsf,migrationsf_inv,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call DMGetSection(distributedMesh,section,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call
>>>         PetscSectionCreate(Petsc_Comm_World,section_seq,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call
>>>         PetscSFDistributeSection(migrationsf_inv,section, &
>>>         PETSC_NULL_INTEGER,section_seq,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call
>>>         DMPlexCreateGlobalToNaturalSF(distributedMesh, &
>>>         section_seq,migrationsf,sf_natural,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call
>>>         DMSetUseNatural(distributedMesh,PETSC_TRUE,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call PetscSFDestroy(migrationsf,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ??????????? call PetscSFDestroy(migrationsf_inv,ierr)
>>>         ??????????? CHKERRQ(ierr)
>>>
>>>         ????????? end if
>>>
>>>         Thanks,
>>>
>>>         Danyang
>>>
>>>>
>>>>         ? Thanks,
>>>>
>>>>         ? ? ?Matt
>>>>
>>>>             Regards,
>>>>
>>>>             Danyang
>>>>
>>>>             On 2018-12-03 5:22 a.m., Matthew Knepley wrote:
>>>>>             I need to write a custom Fortran stub for this one. I
>>>>>             will get it done as soon as?possible.
>>>>>
>>>>>             ? Thanks,
>>>>>
>>>>>             ? ? Matt
>>>>>
>>>>>             On Sat, Dec 1, 2018 at 7:16 PM Danyang Su via
>>>>>             petsc-users <petsc-users at mcs.anl.gov
>>>>>             <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>>>
>>>>>                 Hi All,
>>>>>
>>>>>                 I got a simple compilation error when use
>>>>>                 PetscSFDistributeSection in
>>>>>                 Fortran. It looks like the required head files are
>>>>>                 included and the
>>>>>                 parameters are correctly defined. However, when
>>>>>                 compile the code, I got
>>>>>                 error undefined reference to
>>>>>                 `petscsfdistributesection_'. The code is
>>>>>                 shown below. Did I miss anything here?
>>>>>
>>>>>                 #include <petsc/finclude/petscsys.h>
>>>>>                 #include <petsc/finclude/petscvec.h>
>>>>>                 #include <petsc/finclude/petscdm.h>
>>>>>                 #include <petsc/finclude/petscdmplex.h>
>>>>>                 ? ? ? ?use petscsys
>>>>>                 ? ? ? ?use petscvec
>>>>>                 ? ? ? ?use petscdm
>>>>>                 ? ? ? ?use petscdmplex
>>>>>
>>>>>                 ? ? ? ?implicit none
>>>>>
>>>>>                 ? ? ? ?PetscSection :: section, section_seq
>>>>>                 ? ? ? ?PetscSF :: migrationsf_inv, sf_natural
>>>>>                 ? ? ? ?Vec :: vec_global, vec_natural
>>>>>                 ? ? ? ?PetscErrorCode :: ierr
>>>>>
>>>>>                 ? ? ? ?...
>>>>>
>>>>>                 ? ? ? ?call
>>>>>                 PetscSFDistributeSection(migrationsf_inv,section,
>>>>>                 ? ? ? ? ?&
>>>>>                 PETSC_NULL_INTEGER,section_seq,ierr)
>>>>>                 ? ? ? ?CHKERRQ(ierr)
>>>>>
>>>>>
>>>>>                 ? ? ? ? ? ?call
>>>>>                 PetscSFDistributeSection(migrationsf_inv,section,
>>>>>                 ? ? ?&
>>>>>                 ?PETSC_NULL_INTEGER,section_seq,ierr)
>>>>>                 ? ? ? ? ? ?CHKERRQ(ierr)
>>>>>
>>>>>                 Thanks,
>>>>>
>>>>>                 Danyang
>>>>>
>>>>>
>>>>>
>>>>>             -- 
>>>>>             What most experimenters take for granted before they
>>>>>             begin their experiments is infinitely more interesting
>>>>>             than any results to which their experiments lead.
>>>>>             -- Norbert Wiener
>>>>>
>>>>>             https://www.cse.buffalo.edu/~knepley/
>>>>>             <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>>
>>>>
>>>>         -- 
>>>>         What most experimenters take for granted before they begin
>>>>         their experiments is infinitely more interesting than any
>>>>         results to which their experiments lead.
>>>>         -- Norbert Wiener
>>>>
>>>>         https://www.cse.buffalo.edu/~knepley/
>>>>         <http://www.cse.buffalo.edu/~knepley/>
>>>
>>>
>>>
>>>     -- 
>>>     What most experimenters take for granted before they begin their
>>>     experiments is infinitely more interesting than any results to
>>>     which their experiments lead.
>>>     -- Norbert Wiener
>>>
>>>     https://www.cse.buffalo.edu/~knepley/
>>>     <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/ac758c00/attachment-0001.html>
-------------- next part --------------
#PETSc variables for development version, version V3.6.0 and later
include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

FFLAGS = 

OBJS = ./mesh_data.o\
	./natural.o

natural: $(OBJS) chkopts
	-${FLINKER} $(FFLAGS) -o natural $(OBJS) ${PETSC_LIB}
	${RM} $(SRC)*.o $(SRC)*.mod

-------------- next part --------------
# vtk DataFile Version 2.0
mesh-2d, Created by Gmsh
ASCII
DATASET UNSTRUCTURED_GRID
POINTS 69 double
0 0 0
3 0 0
3 0 2
0 0 2
0.2 0 0.8
0.4 0 0.8
0.8 0 0.8
1.4 0 0.8
2.2 0 0.8
0.8 0 0.6
0.8 0 1
0.8 0 1.2
0.499999999997822 0 0
0.9999999999960245 0 0
1.499999999994285 0 0
1.999999999996158 0 0
2.499999999998079 0 0
0 0 1.500000000000693
0 0 1.000000000004118
0 0 0.5000000000020808
2.500000000003167 0 2
2.000000000001941 0 2
1.500000000004168 0 2
1.00000000000281 0 2
0.5000000000014051 0 2
3 0 0.4999999999988241
3 0 0.999999999997388
3 0 1.499999999998683
1.799999999998915 0 0.8
1.099999999999501 0 0.8
0.3666666666682681 0 1.233333333331731
1.271751539294908 0 1.355799542686996
2.00985325291648 0 1.455472832778138
0.4741111111103676 0 0.3178750000004925
1.034296721683984 0 1.092713954025319
0.6117161716167377 0 1.58564356435608
0.990728609969793 0 1.559942023676578
1.182424242422426 0 0.2962202380952292
2.477293926290137 0 1.244553762494937
0.493402650403436 0 1.055459773692982
0.5909632735517151 0 0.6476526035870375
1.580443401994861 0 0.4189257391804766
1.612090632780561 0 1.126835640176751
2.59236745238114 0 0.4103840906174537
0.7941638542849001 0 0.3147114254793752
0.2731748474417706 0 1.020420503636702
1.966057517364436 0 0.4673319812431365
0.631702026375183 0 0.9486426017917475
0.3558583280051764 0 0.545353879769248
1.614531084604252 0 1.608669168454826
2.604360529456519 0 0.8445703766706929
0.622744504707201 0 1.257570097567692
2.329925884063499 0 1.676684671728572
0.1816715026262627 0 1.240054510318584
1.27739349359455 0 1.715198584350745
0.6030193133904318 0 0.4851185817672306
1.342971052691814 0 1.056932407471012
0.3067601705739227 0 1.536378567870203
0.8018251367042398 0 1.384587091501278
2.019847562397219 0 1.085372447089965
0.9839363972274676 0 1.319779771906041
0.9335741804208713 0 0.9231784885063297
0.981731303510901 0 0.6560746918499005
0.2238250266085547 0 0.2618395980849577
0.6651074814724786 0 1.087753235265755
2.661443962071361 0 1.684247686844439
0.9932946352142333 0 0.4897210062248542
2.249844184832246 0 0.3177341670352057
1.208153775852939 0 0.581598675699766

CELLS 157 575
1 0
1 1
1 2
1 3
1 4
1 5
1 6
1 7
1 8
1 9
1 10
1 11
2 0 12
2 12 13
2 13 14
2 14 15
2 15 16
2 16 1
2 3 17
2 17 18
2 18 19
2 19 0
2 2 20
2 20 21
2 21 22
2 22 23
2 23 24
2 24 3
2 1 25
2 25 26
2 26 27
2 27 2
2 7 28
2 28 8
2 6 29
2 29 7
2 5 6
2 4 5
2 6 9
2 10 6
2 11 10
3 5 48 40
3 35 30 51
3 23 35 36
3 10 34 11
3 40 48 55
3 1 25 43
3 1 43 16
3 10 64 47
3 8 38 59
3 31 42 49
3 21 49 32
3 23 24 35
3 6 10 47
3 4 45 18
3 21 32 52
3 6 29 61
3 6 62 29
3 4 19 48
3 31 60 34
3 7 41 28
3 32 49 42
3 46 67 8
3 32 59 38
3 15 46 41
3 3 57 24
3 39 47 64
3 31 34 56
3 67 43 8
3 5 47 39
3 26 27 38
3 6 40 9
3 14 15 41
3 0 63 19
3 5 39 45
3 8 50 38
3 10 61 34
3 14 41 37
3 7 68 41
3 7 28 42
3 9 55 44
3 35 51 58
3 13 14 37
3 9 44 66
3 4 5 45
3 57 30 35
3 24 57 35
3 32 38 52
3 3 17 57
3 29 56 34
3 25 26 50
3 25 50 43
3 7 56 29
3 35 58 36
3 37 66 44
3 4 48 5
3 26 38 50
3 32 42 59
3 31 36 60
3 27 65 38
3 37 41 68
3 11 34 60
3 33 55 48
3 28 41 46
3 8 28 46
3 20 21 52
3 30 57 53
3 8 43 50
3 21 22 49
3 11 60 58
3 7 42 56
3 62 66 68
3 12 13 44
3 23 36 54
3 9 40 55
3 30 45 39
3 29 34 61
3 19 63 48
3 13 37 44
3 0 12 63
3 28 59 42
3 31 49 54
3 31 56 42
3 18 45 53
3 29 62 68
3 31 54 36
3 12 44 33
3 22 23 54
3 12 33 63
3 2 65 27
3 2 20 65
3 33 48 63
3 30 39 51
3 33 44 55
3 15 16 67
3 38 65 52
3 8 59 28
3 11 58 51
3 6 9 62
3 6 61 10
3 10 11 64
3 16 43 67
3 17 53 57
3 36 58 60
3 15 67 46
3 7 29 68
3 11 51 64
3 22 54 49
3 9 66 62
3 39 64 51
3 20 52 65
3 30 53 45
3 37 68 66
3 19 4 18
3 5 40 6
3 5 6 47
3 17 18 53

CELL_TYPES 157
1
1
1
1
1
1
1
1
1
1
1
1
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mesh_data.F90
Type: text/x-fortran
Size: 6085 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/ac758c00/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: natural.F90
Type: text/x-fortran
Size: 12372 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/ac758c00/attachment-0003.bin>

From mbuerkle at web.de  Thu Feb 28 20:16:29 2019
From: mbuerkle at web.de (Marius Buerkle)
Date: Fri, 1 Mar 2019 03:16:29 +0100
Subject: [petsc-users] Nonconforming object sizes in axpy
In-Reply-To: <366821B4-18D5-4305-8206-DA2CC59CA6E9@gmail.com>
References: <trinity-907740cc-4c1b-4c2e-92b4-d4e25e4ada64-1551103247140@3c-app-webde-bap17>
	<CAMYG4GmtHh6TEuGohPAx1u96C55RnnzewtHUCe2gExxWdqMLNw@mail.gmail.com>
	<trinity-5b03409f-ace7-41bc-99bc-595cb73a04a3-1551106684025@3c-app-webde-bap17>
	<CAGPUisgMSR1u5p4jB445avY6xQk2D_8Lmn7dPc110yhAN105og@mail.gmail.com>
	<trinity-11dd17f9-9c54-4549-a635-6683a2c18c59-1551160302158@3c-app-webde-bap57>
	<59CE362D-3C22-48E3-9ECB-ACBA0E6BAB4B@gmail.com>
	<trinity-ae99b9e9-7bf9-4e2a-81aa-21bde5ba7558-1551163843895@3c-app-webde-bap57>
	<99CA20EC-8E26-4F2B-B971-BB210C8DCF65@gmail.com>
	<trinity-1fbea6d6-9f25-4efb-b101-d0c6671d58ad-1551309918539@3c-app-webde-bs40>
	<7183BE65-3FA8-46BA-8DEA-612E82AD4878@gmail.com>
	<trinity-be8a5288-1b29-4aac-a004-929713d6add4-1551311952389@3c-app-webde-bap04>
	<366821B4-18D5-4305-8206-DA2CC59CA6E9@gmail.com>
Message-ID: <trinity-e67c2774-e9bc-4f76-baff-f15891398382-1551406589550@3c-app-webde-bs29>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190301/b1787dbd/attachment.html>