<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Matt,
<div class=""><br class="">
</div>
<div class="">Could you issue MR to get changes from knepley/fix-dm-g2n-serial into PETSc master? I’m having some trouble during PETSc installation withexodusii using your branch, but the master seems fine.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">-Gautam.</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 18, 2020, at 11:19 AM, 'Bisht, Gautam' via tdycores-dev <<a href="mailto:tdycores-dev@googlegroups.com" class="">tdycores-dev@googlegroups.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Hi Matt,</div>
<div class=""><br class="">
</div>
<div class="">Thanks for the fixes to the example.</div>
<div class=""><br class="">
</div>
<div class="">-Gautam<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 15, 2020, at 7:05 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div dir="ltr" class="">On Wed, Jan 15, 2020 at 4:08 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr" class="">
<div dir="ltr" class="">On Wed, Jan 15, 2020 at 3:47 PM 'Bisht, Gautam' via tdycores-dev <<a href="mailto:tdycores-dev@googlegroups.com" target="_blank" class="">tdycores-dev@googlegroups.com</a>> wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div class="">
<div style="word-wrap: break-word;" class="">
<div class="">Hi Matt,</div>
<div class=""><br class="">
</div>
<div class="">I’m running into error while using DMPlexNaturalToGlobalBegin/End and am hoping you have some insights in what I’m doing incorrectly. I create a 2x2x2 grid and distribute it across processors (N=1,2). I create a natural and a global vector; and
then call DMPlexNaturalToGlobalBegin/End. Here are the two issues:</div>
<div class=""><br class="">
</div>
<div class="">- When N = 1, PETSc complains about DMSetUseNatural() not being called before DMPlexDistribute(), which is certainly not the case.</div>
<div class="">- For N=1 and 2, global vector doesn’t have valid entries.</div>
<div class=""><br class="">
</div>
<div class="">I’m not sure how to create the natural vector and have used DMCreateGlobalVector() to create the natural vector, which could be the issue.</div>
<div class=""><br class="">
</div>
<div class="">Attached is the sample code to reproduce the error and below is the screen output.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Cool. I will run it and figure out the problem.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">1) There was bad error reporting there. I am putting the fix in a new branch. It did not check for being on one process. If you run with</div>
<div class=""><br class="">
</div>
<div class=""> knepley/fix-dm-g2n-serial</div>
<div class=""><br class="">
</div>
<div class=""> It will work correctly in serial.</div>
<div class=""><br class="">
</div>
<div class="">2) The G2N needs a serial data layout to work, so you have to make a Section _before_ distributing. I need to put that in the docs. I have</div>
<div class=""> fixed your example to do this and attached it. I run it with</div>
<div class=""><br class="">
</div>
<div class=""> <span style="font-family: Menlo; font-size: 11px;" class="">master *:~/Downloads/tmp/Gautam$ /PETSc3/petsc/bin/mpiexec -n 1 ./ex_test -dm_plex_box_faces 2,2,2 -dm_view</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
DM Object: 1 MPI processes</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: plex</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
DM_0x84000000_0 in 3 dimensions:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>0-cells: 27</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>1-cells: 54</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>2-cells: 36</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>3-cells: 8</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Labels:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>marker: 1 strata with value/size (1 (72))</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>Face Sets: 6 strata with value/size (6 (4), 5 (4), 3 (4), 4 (4), 1 (4), 2 (4))</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Field p:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>adjacency FVM++</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Natural vector:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Vec Object: 1 MPI processes</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: seq</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
0.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
1.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
3.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
4.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
5.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
6.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
7.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Global vector:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Vec Object: 1 MPI processes</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: seq</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
0.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
1.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
3.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
4.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
5.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
6.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
7.</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Information about the mesh:</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 00; (0.250000, 0.250000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 01; (0.750000, 0.250000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 02; (0.250000, 0.750000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 03; (0.750000, 0.750000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 04; (0.250000, 0.250000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 05; (0.750000, 0.250000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 06; (0.250000, 0.750000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 07; (0.750000, 0.750000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<br class="">
</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
master *:~/Downloads/tmp/Gautam$ /PETSc3/petsc/bin/mpiexec -n 2 ./ex_test -dm_plex_box_faces 2,2,2 -dm_view</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
DM Object: Parallel Mesh 2 MPI processes</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: plex</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Parallel Mesh in 3 dimensions:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>0-cells: 27 27</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>1-cells: 54 54</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>2-cells: 36 36</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>3-cells: 8 8</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Labels:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>marker: 1 strata with value/size (1 (72))</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), 6 (4))</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Field p:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>adjacency FVM++</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Natural vector:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Vec Object: 2 MPI processes</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: mpi</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Process [0]</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
0.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
1.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
3.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Process [1]</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
4.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
5.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
6.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
7.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Global vector:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Vec Object: 2 MPI processes</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-Apple-converted-space"> <span class="Apple-converted-space"> </span></span>type: mpi</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Process [0]</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
2.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
3.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
6.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
7.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Process [1]</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
0.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
1.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
4.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
5.</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class="">
<br class="">
</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
Information about the mesh:</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 00; (0.250000, 0.750000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 01; (0.750000, 0.750000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 02; (0.250000, 0.750000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 03; (0.750000, 0.750000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 04; (0.250000, 0.250000, 0.250000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 05; (0.750000, 0.250000, 0.250000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 06; (0.250000, 0.250000, 0.750000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[0] cell = 07; (0.750000, 0.250000, 0.750000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 00; (0.250000, 0.250000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 01; (0.750000, 0.250000, 0.250000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 02; (0.250000, 0.250000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 03; (0.750000, 0.250000, 0.750000); is_local = 1</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 04; (0.250000, 0.750000, 0.250000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 05; (0.750000, 0.750000, 0.250000); is_local = 0</div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 06; (0.250000, 0.750000, 0.750000); is_local = 0</div>
<p class="gmail-p1" style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;">
</p>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class="">
[1] cell = 07; (0.750000, 0.750000, 0.750000); is_local = 0</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div class="">
<div style="word-wrap: break-word;" class="">
<div class="">>make ex_test</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</div>
<div class="">>$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 1 ./ex_test</div>
<div class="">Natural vector:</div>
<div class=""><br class="">
</div>
<div class="">Vec Object: 1 MPI processes</div>
<div class=""> type: seq</div>
<div class="">0.</div>
<div class="">1.</div>
<div class="">2.</div>
<div class="">3.</div>
<div class="">4.</div>
<div class="">5.</div>
<div class="">6.</div>
<div class="">7.</div>
<div class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div>
<div class="">[0]PETSC ERROR: Object is in wrong state</div>
<div class="">[0]PETSC ERROR: DM global to natural SF was not created.</div>
<div class="">You must call DMSetUseNatural() before DMPlexDistribute().</div>
<div class=""><br class="">
</div>
<div class="">[0]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://protect2.fireeye.com/v1/url?k=49ab253e-151e1a87-49ab0f2b-0cc47adc5fce-24c0c8c46a909b39&q=1&e=e8c1de75-1fa2-4cdd-a1a0-86a8a8811e0a&u=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a><span class="Apple-converted-space"> </span>for
trouble shooting.</div>
<div class="">[0]PETSC ERROR: Petsc Development GIT revision: v3.12.2-537-g5f77d1e0e5 GIT Date: 2019-12-21 14:33:27 -0600</div>
<div class="">[0]PETSC ERROR: ./ex_test on a darwin-gcc8 named WE37411 by bish218 Wed Jan 15 12:34:03 2020</div>
<div class="">[0]PETSC ERROR: Configure options --with-blaslapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-hdf5-dir=/opt/local --download-zlib --download-exodusii=yes
--download-hdf5=yes --download-netcdf=yes --download-pnetcdf=yes --download-hypre=yes --download-mpich=yes --download-mumps=yes --download-scalapack=yes --with-cc=/opt/local/bin/gcc-mp-8 --with-cxx=/opt/local/bin/g++-mp-8 --with-fc=/opt/local/bin/gfortran-mp-8
--download-sowing=1 PETSC_ARCH=darwin-gcc8</div>
<div class="">[0]PETSC ERROR: #1 DMPlexNaturalToGlobalBegin() line 289 in /Users/bish218/projects/petsc/petsc_v3.12.2/src/dm/impls/plex/plexnatural.c</div>
<div class=""><br class="">
</div>
<div class="">Global vector:</div>
<div class=""><br class="">
</div>
<div class="">Vec Object: 1 MPI processes</div>
<div class=""> type: seq</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class=""><br class="">
</div>
<div class="">Information about the mesh:</div>
<div class=""><br class="">
</div>
<div class="">Rank = 0</div>
<div class="">local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1</div>
<div class="">local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1</div>
<div class="">local_id = 02; (0.250000, 0.750000, 0.250000); is_local = 1</div>
<div class="">local_id = 03; (0.750000, 0.750000, 0.250000); is_local = 1</div>
<div class="">local_id = 04; (0.250000, 0.250000, 0.750000); is_local = 1</div>
<div class="">local_id = 05; (0.750000, 0.250000, 0.750000); is_local = 1</div>
<div class="">local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 1</div>
<div class="">local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 1</div>
</div>
<div class=""><br class="">
</div>
<div class="">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</div>
<div class=""><br class="">
</div>
<div class="">>$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 2 ./ex_test</div>
<div class="">Natural vector:</div>
<div class=""><br class="">
</div>
<div class="">Vec Object: 2 MPI processes</div>
<div class=""> type: mpi</div>
<div class="">Process [0]</div>
<div class="">0.</div>
<div class="">1.</div>
<div class="">2.</div>
<div class="">3.</div>
<div class="">Process [1]</div>
<div class="">4.</div>
<div class="">5.</div>
<div class="">6.</div>
<div class="">7.</div>
<div class=""><br class="">
</div>
<div class="">Global vector:</div>
<div class=""><br class="">
</div>
<div class="">Vec Object: 2 MPI processes</div>
<div class=""> type: mpi</div>
<div class="">Process [0]</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">Process [1]</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class="">0.</div>
<div class=""><br class="">
</div>
<div class="">Information about the mesh:</div>
<div class=""><br class="">
</div>
<div class="">Rank = 0</div>
<div class="">local_id = 00; (0.250000, 0.750000, 0.250000); is_local = 1</div>
<div class="">local_id = 01; (0.750000, 0.750000, 0.250000); is_local = 1</div>
<div class="">local_id = 02; (0.250000, 0.750000, 0.750000); is_local = 1</div>
<div class="">local_id = 03; (0.750000, 0.750000, 0.750000); is_local = 1</div>
<div class="">local_id = 04; (0.250000, 0.250000, 0.250000); is_local = 0</div>
<div class="">local_id = 05; (0.750000, 0.250000, 0.250000); is_local = 0</div>
<div class="">local_id = 06; (0.250000, 0.250000, 0.750000); is_local = 0</div>
<div class="">local_id = 07; (0.750000, 0.250000, 0.750000); is_local = 0</div>
<div class=""><br class="">
</div>
<div class="">Rank = 1</div>
<div class="">local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1</div>
<div class="">local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1</div>
<div class="">local_id = 02; (0.250000, 0.250000, 0.750000); is_local = 1</div>
<div class="">local_id = 03; (0.750000, 0.250000, 0.750000); is_local = 1</div>
<div class="">local_id = 04; (0.250000, 0.750000, 0.250000); is_local = 0</div>
<div class="">local_id = 05; (0.750000, 0.750000, 0.250000); is_local = 0</div>
<div class="">local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 0</div>
<div class="">local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 0</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">-Gautam</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div style="word-wrap: break-word;" class=""></div>
<div style="word-wrap: break-word;" class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 9, 2020, at 4:57 PM, 'Bisht, Gautam' via tdycores-dev <<a href="mailto:tdycores-dev@googlegroups.com" target="_blank" class="">tdycores-dev@googlegroups.com</a>> wrote:</div>
<br class="">
<div class="">
<div style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<br class="">
<br class="">
<blockquote type="cite" class="">
<div class="">On Jan 9, 2020, at 4:25 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="ltr" class="">On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev <<a href="mailto:tdycores-dev@googlegroups.com" target="_blank" class="">tdycores-dev@googlegroups.com</a>> wrote:<br class="">
</div>
<div class="">
<blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;" class="">
<br class="">
> On Jan 9, 2020, at 2:58 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank" class="">jed@jedbrown.org</a>> wrote:<br class="">
><span class=""> </span><br class="">
> "'Bisht, Gautam' via tdycores-dev" <<a href="mailto:tdycores-dev@googlegroups.com" target="_blank" class="">tdycores-dev@googlegroups.com</a>> writes:<br class="">
><span class=""> </span><br class="">
>>> Do you need to rely on the element number, or would coordinates (of a<br class="">
>>> centroid?) be sufficient for your purposes? <span class=""> </span><br class="">
>><span class=""> </span><br class="">
>> I do need to rely on the element number. In my case, I have a mapping file that remaps data from one grid onto another grid. Though I’m currently creating a hexahedron mesh, in the future I would be reading in an unstructured grid from a file for which
I cannot rely on coordinates.<br class="">
><span class=""> </span><br class="">
> How does the mapping file work and how is it generated?<br class="">
<br class="">
In CESM/E3SM, the mapping file is used to map fluxes or states between grids of two components (e.g. land & atmosphere). The mapping method can be conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses ESMF_RegridWeightGen to generate the mapping
file, I’m using by own MATLAB script to create the mapping file.<br class="">
<br class="">
I’m surprised that this is not an issue for other codes that are using DMPlex. E.g In PFLOTRAN, when a user creates a custom unstructured grid, they can specify material property for each grid cell. So, there should be a way to create a vectorscatter that will
scatter material property read in the “application”-order (i.e. order before calling DMPlexDistribute() ) to ghosted-order (i.e. order after calling DMPlexDistribute()).<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">We did build something specific for this because some people wanted it. I wish I could purge this from all simulations. Its </div>
<div class="">definitely destructive, but this is the way the world currently is.</div>
<div class=""><br class="">
</div>
<div class="">You want this:</div>
<div class=""><br class="">
</div>
<div class=""> <a href="https://protect2.fireeye.com/v1/url?k=5fd6bd0b-036383c4-5fd6971e-0cc47adc5e60-2156f2075f6da02c&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fpetsc-current%2Fdocs%2Fmanualpages%2FDMPLEX%2FDMPlexNaturalToGlobalBegin.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html</a></div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Perfect.</div>
<div class=""><br class="">
</div>
<div class="">Thanks.</div>
<div class="">-Gautam</div>
<div class=""><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div class="">
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;" class="">
> We can locate points and create interpolation with unstructured grids.<br class="">
><span class=""> </span><br class="">
> --<span class=""> </span><br class="">
> You received this message because you are subscribed to the Google Groups "tdycores-dev" group.<br class="">
> To unsubscribe from this group and stop receiving emails from it, send an email to<span class=""> </span><a href="mailto:tdycores-dev%2Bunsubscribe@googlegroups.com" target="_blank" class="">tdycores-dev+unsubscribe@googlegroups.com</a>.<br class="">
> To view this discussion on the web visit<span class=""> </span><a href="https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4&q=1&e=0962f8e1-9155-4d9c-abdf-2b6481141cd0&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org" rel="noreferrer" target="_blank" class="">https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4&q=1&e=0962f8e1-9155-4d9c-abdf-2b6481141cd0&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org</a>.<br class="">
<br class="">
--<span class=""> </span><br class="">
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.<br class="">
To unsubscribe from this group and stop receiving emails from it, send an email to<span class=""> </span><a href="mailto:tdycores-dev%2Bunsubscribe@googlegroups.com" target="_blank" class="">tdycores-dev+unsubscribe@googlegroups.com</a>.<br class="">
To view this discussion on the web visit<span class=""> </span><a href="https://protect2.fireeye.com/v1/url?k=711264f2-2da75a3d-71124ee7-0cc47adc5e60-a1731ca51d9b5203&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F9AB001AF-8857-446A-AE69-E8D6A25CB8FA%2540pnnl.gov" rel="noreferrer" target="_blank" class="">https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov</a>.<br class="">
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class=""> </span><br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<br class="">
</div>
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">--<span class=""> </span></span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">You
received this message because you are subscribed to the Google Groups "tdycores-dev" group.</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">To
unsubscribe from this group and stop receiving emails from it, send an email to<span class=""> </span></span><a href="mailto:tdycores-dev+unsubscribe@googlegroups.com" target="_blank" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">tdycores-dev+unsubscribe@googlegroups.com</a><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">.</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">To
view this discussion on the web visit<span class=""> </span></span><a href="https://protect2.fireeye.com/v1/url?k=ce43b754-92f6899b-ce439d41-0cc47adc5e60-0a864c0f80b4c75d&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2FCAMYG4Gm%253DSY%253DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%252Bv-zVvFnA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter" target="_blank" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gm%3DSY%3DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%2Bv-zVvFnA%40mail.gmail.com</a><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">.</span></div>
</blockquote>
</div>
<br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<br class="">
</div>
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">--<span class=""> </span></span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">You
received this message because you are subscribed to the Google Groups "tdycores-dev" group.</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">To
unsubscribe from this group and stop receiving emails from it, send an email to<span class=""> </span></span><a href="mailto:tdycores-dev+unsubscribe@googlegroups.com" target="_blank" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">tdycores-dev+unsubscribe@googlegroups.com</a><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">.</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">To
view this discussion on the web visit<span class=""> </span></span><a href="https://protect2.fireeye.com/v1/url?k=365c6512-6ae95bdd-365c4f07-0cc47adc5e60-f9b3b6f5e8918ac5&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F7C23ABBA-2F76-4EAB-9834-9391AD77E18B%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter" target="_blank" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">https://groups.google.com/d/msgid/tdycores-dev/7C23ABBA-2F76-4EAB-9834-9391AD77E18B%40pnnl.gov</a><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline;" class="">.</span></div>
</blockquote>
</div>
<br class="">
</div>
</div>
<div class=""><br class="webkit-block-placeholder">
</div>
--<span class="Apple-converted-space"> </span><br class="">
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.<br class="">
To unsubscribe from this group and stop receiving emails from it, send an email to<span class="Apple-converted-space"> </span><a href="mailto:tdycores-dev+unsubscribe@googlegroups.com" target="_blank" class="">tdycores-dev+unsubscribe@googlegroups.com</a>.<br class="">
To view this discussion on the web visit<span class="Apple-converted-space"> </span><a href="https://protect2.fireeye.com/v1/url?k=d941301d-85f40fa4-d9411a08-0cc47adc5fce-ae3accbb18dd8e05&q=1&e=e8c1de75-1fa2-4cdd-a1a0-86a8a8811e0a&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8A7925AE-08F5-4F81-AAA5-B2FDC3D833B0%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter" target="_blank" class="">https://groups.google.com/d/msgid/tdycores-dev/8A7925AE-08F5-4F81-AAA5-B2FDC3D833B0%40pnnl.gov</a>.<br class="">
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div dir="ltr" class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="webkit-block-placeholder">
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You
received this message because you are subscribed to the Google Groups "tdycores-dev" group.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">To
unsubscribe from this group and stop receiving emails from it, send an email to<span class="Apple-converted-space"> </span></span><a href="mailto:tdycores-dev+unsubscribe@googlegroups.com" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">tdycores-dev+unsubscribe@googlegroups.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">To
view this discussion on the web visit<span class="Apple-converted-space"> </span></span><a href="https://protect2.fireeye.com/v1/url?k=bae87786-e65d483f-bae85d93-0cc47adc5fce-c50576531e121267&q=1&e=e8c1de75-1fa2-4cdd-a1a0-86a8a8811e0a&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2FCAMYG4Gn%253DxsVjjN8sX6km8ub%253Djkk8vxiU2DZVEi-4Kpbi_rM-0w%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter" style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gn%3DxsVjjN8sX6km8ub%3Djkk8vxiU2DZVEi-4Kpbi_rM-0w%40mail.gmail.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span id="cid:f_k5g5gy550" class=""><ex_test.c></span></div>
</blockquote>
</div>
<br class="">
</div>
</div>
<div class=""><br class="webkit-block-placeholder">
</div>
-- <br class="">
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.<br class="">
To unsubscribe from this group and stop receiving emails from it, send an email to
<a href="mailto:tdycores-dev+unsubscribe@googlegroups.com" class="">tdycores-dev+unsubscribe@googlegroups.com</a>.<br class="">
To view this discussion on the web visit <a href="https://protect2.fireeye.com/v1/url?k=24802983-7835163a-24800396-0cc47adc5fce-dc4bfcfa4b93062d&q=1&e=d8ec9e84-40ee-4fcf-8e21-5e91bd34392f&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F3F79926B-D567-4592-8E4E-46D21628D2DF%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter" class="">
https://groups.google.com/d/msgid/tdycores-dev/3F79926B-D567-4592-8E4E-46D21628D2DF%40pnnl.gov</a>.<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>