<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Mark, <br>
<br>
Thanks for the information. <br>
<br>
@Junchao: Given that there are known issues with GPU aware MPI, it might be best to wait until there is an updated version of cray-mpich (which hopefully contains the relevant fixes).<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank You,<br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="font-size:12.8px">Sajid Ali (he/him) | Research Associate<br>
</div>
<div style="font-size:12.8px">Scientific Computing Division<br>
</div>
<div style="font-size:12.8px">Fermi National Accelerator Laboratory<br>
</div>
<span style="font-size:12.8px"><a href="http://s-sajid-ali.github.io" target="_blank">s-sajid-ali.github.io</a></span></div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mark Adams <mfadams@lbl.gov><br>
<b>Sent:</b> Thursday, February 10, 2022 8:47 PM<br>
<b>To:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Cc:</b> Sajid Ali Syed <sasyed@fnal.gov>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] GAMG crash during setup when using multiple GPUs</font>
<div> </div>
</div>
<div>
<div dir="ltr">Perlmutter has problems with GPU aware MPI.
<div>This is being actively worked on at NERSc.</div>
<div><br>
</div>
<div>Mark</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Feb 10, 2022 at 9:22 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>Hi, Sajid Ali,</div>
<div> I have no clue. I have access to perlmutter. I am thinking how to debug that. </div>
<div> If your app is open-sourced and easy to build, then I can build and debug it. Otherwise, suppose you build and install petsc (only with options needed by your app) to a shared directory, and I can access your executable (which uses RPATH for libraries),
then maybe I can debug it (I only need to install my own petsc to the shared directory)</div>
<div><br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Feb 10, 2022 at 6:04 PM Sajid Ali Syed <<a href="mailto:sasyed@fnal.gov" target="_blank">sasyed@fnal.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Junchao, <br>
<br>
With "-use_gpu_aware_mpi 0" there is no error. I'm attaching the log for this case with this email.
<br>
<br>
I also ran with gpu aware mpi to see if I could reproduce the error and got the error but from a different location. This logfile is also attached.
<br>
<br>
This was using the newest cray-mpich on NERSC-perlmutter (8.1.12). Let me know if I can share further information to help with debugging this.
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_gmail-m_-8046890637211492674gmail-m_-1165435411634617833Signature">
<div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thank You,<br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="font-size:12.8px">Sajid Ali (he/him) | Research Associate<br>
</div>
<div style="font-size:12.8px">Scientific Computing Division<br>
</div>
<div style="font-size:12.8px">Fermi National Accelerator Laboratory<br>
</div>
<span style="font-size:12.8px"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=Fea4VIbc4UoqdTFjAk3kg3Hp94LYXkjR3gHIdP08lMeT-3zEDZNKDcHjRejBIggW&s=ezCw13eIYUcCzUki3rlnpGZWZrdcTxlGpG57GqrEz_s&e=" target="_blank">s-sajid-ali.github.io</a></span></div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<div id="x_gmail-m_-8046890637211492674gmail-m_-1165435411634617833appendonsend">
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-8046890637211492674gmail-m_-1165435411634617833divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Thursday, February 10, 2022 1:43 PM<br>
<b>To:</b> Sajid Ali Syed <<a href="mailto:sasyed@fnal.gov" target="_blank">sasyed@fnal.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] GAMG crash during setup when using multiple GPUs</font>
<div> </div>
</div>
<div>
<div dir="ltr">Also, try "-use_gpu_aware_mpi 0" to see if there is a difference.
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Feb 10, 2022 at 1:40 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>Did it fail without GPU at 64 MPI ranks?</div>
<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div>
<div dir="ltr">On Thu, Feb 10, 2022 at 1:22 PM Sajid Ali Syed <<a href="mailto:sasyed@fnal.gov" target="_blank">sasyed@fnal.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>
<p style="margin:0px 0px 1.2em">Hi PETSc-developers, </p>
<p style="margin:0px 0px 1.2em">I’m seeing the following crash that occurs during the setup phase of the preconditioner when using multiple GPUs. The relevant error trace is shown below:
</p>
<pre style="font-family:Consolas,Inconsolata,Courier,monospace; font-size:1em; line-height:1.2em; margin:1.2em 0px"><code style="font-size:0.85em; font-family:Consolas,Inconsolata,Courier,monospace; margin:0px 0.15em; background-color:rgb(248,248,248); white-space:pre-wrap; overflow:auto; border-radius:3px; border:1px solid rgb(204,204,204); padding:0.5em 0.7em; display:block">(GTL DEBUG: 26) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272<br>[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: General MPI error
[24]PETSC ERROR: MPI error 1 Invalid buffer pointer
[24]PETSC ERROR: See <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__petsc.org_release_faq_&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=3AFKDE-HT__MEeFxdxlc6bMDLLjchFccw_htjVmWkOsApaEairnUJYnKT28tfsiN&s=ZpvtorGvQdUD8O-wLBTUYUUb6-Kccver8Cc4kXlZ7J0&e=" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.
[24]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae GIT Date: 2022-02-08 15:08:19 +0000
...
[24]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[24]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[24]PETSC ERROR: #3 PetscSFBcastEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:218
[24]PETSC ERROR: #4 PetscSFBcastEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1499
[24]PETSC ERROR: #5 VecScatterEnd_Internal() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:87
[24]PETSC ERROR: #6 VecScatterEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:1366
[24]PETSC ERROR: #7 MatMult_MPIAIJCUSPARSE() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpicusparse/<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mpiaijcusparse.cu-3A302&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=3AFKDE-HT__MEeFxdxlc6bMDLLjchFccw_htjVmWkOsApaEairnUJYnKT28tfsiN&s=eMW4lGCKOn_tzQeT5gnM0i9mgEMwwbOe1EkCAtKG9M8&e=" target="_blank">mpiaijcusparse.cu:302</a>
[24]PETSC ERROR: #8 MatMult() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/interface/matrix.c:2438
[24]PETSC ERROR: #9 PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:730
[24]PETSC ERROR: #10 KSP_PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/petsc/private/kspimpl.h:421
[24]PETSC ERROR: #11 KSPGMRESCycle() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:162
[24]PETSC ERROR: #12 KSPSolve_GMRES() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:247
[24]PETSC ERROR: #13 KSPSolve_Private() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:925
[24]PETSC ERROR: #14 KSPSolve() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:1103
[24]PETSC ERROR: #15 PCGAMGOptProlongator_AGG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/agg.c:1127
[24]PETSC ERROR: #16 PCSetUp_GAMG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/gamg.c:626
[24]PETSC ERROR: #17 PCSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:1017
[24]PETSC ERROR: #18 KSPSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:417
[24]PETSC ERROR: #19 main() at poisson3d.c:69
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -dm_mat_type aijcusparse
[24]PETSC ERROR: -dm_vec_type cuda
[24]PETSC ERROR: -ksp_monitor
[24]PETSC ERROR: -ksp_norm_type unpreconditioned
[24]PETSC ERROR: -ksp_type cg
[24]PETSC ERROR: -ksp_view
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[24]PETSC ERROR: -mg_levels_ksp_type chebyshev
[24]PETSC ERROR: -mg_levels_pc_type jacobi
[24]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[24]PETSC ERROR: -pc_gamg_square_graph 1
[24]PETSC ERROR: -pc_gamg_threshold 0.0
[24]PETSC ERROR: -pc_gamg_threshold_scale 0.0
[24]PETSC ERROR: -pc_gamg_type agg
[24]PETSC ERROR: -pc_type gamg
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
</code></pre>
<p style="margin:0px 0px 1.2em">Attached with this email is the full error log and the submit script for a 8-node/64-GPU/64 MPI rank job. I’ll also note that the same program did not crash when using either 2 or 4 nodes (with 8 & 16 GPUs/MPI ranks respectively)
and attach those logs as well if that helps. Could someone let me know what this error means and what can be done to prevent it?</p>
<p style="margin:0px 0px 1.2em">Thank You,<br>
Sajid Ali (he/him) | Research Associate</p>
<p style="margin:0px 0px 1.2em">Scientific Computing Division</p>
<p style="margin:0px 0px 1.2em">Fermi National Accelerator Laboratory</p>
<p style="margin:0px 0px 1.2em"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=3AFKDE-HT__MEeFxdxlc6bMDLLjchFccw_htjVmWkOsApaEairnUJYnKT28tfsiN&s=6Fj7FO5IQGRCkPfC22pD7hAo0AxsVgu3kG9LNOftqK0&e=" target="_blank">s-sajid-ali.github.io</a></p>
<div title="MDH:PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IENhbGlicmksIEFyaWFsLCBIZWx2ZXRpY2EsIHNhbnMt
c2VyaWY7IGZvbnQtc2l6ZTogMTJwdDsgY29sb3I6IHJnYigwLCAwLCAwKTsiPkhpIFBFVFNjLWRl
dmVsb3BlcnMsIDxicj48YnI+SSdtIHNlZWluZyB0aGUgZm9sbG93aW5nIGNyYXNoIHRoYXQgb2Nj
dXJzIGR1cmluZyB0aGUgc2V0dXAgcGhhc2Ugb2YgdGhlIHByZWNvbmRpdGlvbmVyIHdoZW4gdXNp
bmcgYSBtdWx0aXBsZSAoJmd0OzQpIG9mIEdQVXMuIFRoZSByZWxldmFudCBlcnJvciB0cmFjZSBp
cyBzaG93biBiZWxvdzogPGJyPjxicj5gYGA8YnI+KEdUTCBERUJVRzogMjYpIGN1SXBjT3Blbk1l
bUhhbmRsZTogcmVzb3VyY2UgYWxyZWFkeSBtYXBwZWQsIENVREFfRVJST1JfQUxSRUFEWV9NQVBQ
RUQsIGxpbmUgbm8gMjcyPGRpdj5bMjRdUEVUU0MgRVJST1I6IC0tLS0tLS0tLS0tLS0tLS0tLS0t
LSBFcnJvciBNZXNzYWdlIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IEdlbmVyYWwg
TVBJIGVycm9yPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IE1QSSBlcnJvciAxIEludmFsaWQg
YnVmZmVyIHBvaW50ZXI8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogU2VlIGh0dHBzOi8vcGV0
c2Mub3JnL3JlbGVhc2UvZmFxLyBmb3IgdHJvdWJsZSBzaG9vdGluZy48L2Rpdj5bMjRdUEVUU0Mg
RVJST1I6IFBldHNjIERldmVsb3BtZW50IEdJVCByZXZpc2lvbjogZjM1MWQ1NDk0YjU0NjJmNjJj
NDE5ZTAwNjQ1YWMyZTQ3N2I4OGNhZSAmbmJzcDtHSVQgRGF0ZTogMjAyMi0wMi0wOCAxNTowODox
OSArMDAwMDxicj4uLi48YnI+WzI0XVBFVFNDIEVSUk9SOiAjMSBQZXRzY1NGTGlua1dhaXRSZXF1
ZXN0c19NUEkoKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1h
aW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy92ZWMvaXMv
c2YvaW1wbHMvYmFzaWMvc2ZtcGkuYzo1NDxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMiBQZXRzY1NG
TGlua0ZpbmlzaENvbW11bmljYXRpb24oKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNr
LXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2st
c3JjL2luY2x1ZGUvLi4vc3JjL3ZlYy9pcy9zZi9pbXBscy9iYXNpYy9zZnBhY2suaDoyNzQ8L2Rp
dj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzMgUGV0c2NTRkJjYXN0RW5kX0Jhc2ljKCkgYXQgL3Rt
cC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJvM2Zp
cGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVjL2lzL3NmL2ltcGxzL2Jhc2ljL3Nm
YmFzaWMuYzoyMTg8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzQgUGV0c2NTRkJjYXN0RW5k
KCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2
a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVjL2lzL3NmL2ludGVy
ZmFjZS9zZi5jOjE0OTk8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzUgVmVjU2NhdHRlckVu
ZF9JbnRlcm5hbCgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3RhZ2UtcGV0c2Mt
bWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMvc3JjL3ZlYy9p
cy9zZi9pbnRlcmZhY2UvdnNjYXQuYzo4NzwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjNiBW
ZWNTY2F0dGVyRW5kKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRz
Yy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVj
L2lzL3NmL2ludGVyZmFjZS92c2NhdC5jOjEzNjY8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjog
IzcgTWF0TXVsdF9NUElBSUpDVVNQQVJTRSgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3Bh
Y2stc3RhZ2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFj
ay1zcmMvc3JjL21hdC9pbXBscy9haWovbXBpL21waWN1c3BhcnNlL21waWFpamN1c3BhcnNlLmN1
OjMwMjwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjOCBNYXRNdWx0KCkgYXQgL3RtcC9zYWpp
ZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hl
eXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvbWF0L2ludGVyZmFjZS9tYXRyaXguYzoyNDM4PC9k
aXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICM5IFBDQXBwbHlCQW9yQUIoKSBhdCAvdG1wL3Nhamlk
L3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5
dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9rc3AvcGMvaW50ZXJmYWNlL3ByZWNvbi5jOjczMDwv
ZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMTAgS1NQX1BDQXBwbHlCQW9yQUIoKSBhdCAvdG1w
L3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlw
ZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL2luY2x1ZGUvcGV0c2MvcHJpdmF0ZS9rc3BpbXBs
Lmg6NDIxPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxMSBLU1BHTVJFU0N5Y2xlKCkgYXQg
L3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJv
M2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3NwL2tzcC9pbXBscy9nbXJlcy9n
bXJlcy5jOjE2MjwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMTIgS1NQU29sdmVfR01SRVMo
KSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZr
YmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9rc3Ava3NwL2ltcGxzL2dt
cmVzL2dtcmVzLmM6MjQ3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxMyBLU1BTb2x2ZV9Q
cml2YXRlKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWlu
LW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3NwL2tzcC9p
bnRlcmZhY2UvaXRmdW5jLmM6OTI1PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxNCBLU1BT
b2x2ZSgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3RhZ2UtcGV0c2MtbWFpbi1t
bmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMvc3JjL2tzcC9rc3AvaW50
ZXJmYWNlL2l0ZnVuYy5jOjExMDM8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzE1IFBDR0FN
R09wdFByb2xvbmdhdG9yX0FHRygpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3Rh
Z2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMv
c3JjL2tzcC9wYy9pbXBscy9nYW1nL2FnZy5jOjExMjc8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogIzE2IFBDU2V0VXBfR0FNRygpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3Rh
Z2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMv
c3JjL2tzcC9wYy9pbXBscy9nYW1nL2dhbWcuYzo2MjY8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogIzE3IFBDU2V0VXAoKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBl
dHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9r
c3AvcGMvaW50ZXJmYWNlL3ByZWNvbi5jOjEwMTc8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjog
IzE4IEtTUFNldFVwKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRz
Yy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3Nw
L2tzcC9pbnRlcmZhY2UvaXRmdW5jLmM6NDE3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMx
OSBtYWluKCkgYXQgcG9pc3NvbjNkLmM6Njk8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogUEVU
U2MgT3B0aW9uIFRhYmxlIGVudHJpZXM6PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IC1kbV9t
YXRfdHlwZSBhaWpjdXNwYXJzZTwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtZG1fdmVjX3R5
cGUgY3VkYTwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAta3NwX21vbml0b3I8L2Rpdj48ZGl2
PlsyNF1QRVRTQyBFUlJPUjogLWtzcF9ub3JtX3R5cGUgdW5wcmVjb25kaXRpb25lZDwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAta3NwX3R5cGUgY2c8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogLWtzcF92aWV3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IC1sb2dfdmlldzwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX2VzdGVpZ19rc3BfdHlwZSBjZzwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX2tzcF90eXBlIGNoZWJ5c2hldjwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX3BjX3R5cGUgamFjb2JpPC9kaXY+PGRpdj5b
MjRdUEVUU0MgRVJST1I6IC1wY19nYW1nX2FnZ19uc21vb3RocyAxPC9kaXY+PGRpdj5bMjRdUEVU
U0MgRVJST1I6IC1wY19nYW1nX3NxdWFyZV9ncmFwaCAxPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJS
T1I6IC1wY19nYW1nX3RocmVzaG9sZCAwLjA8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogLXBj
X2dhbWdfdGhyZXNob2xkX3NjYWxlIDAuMDwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtcGNf
Z2FtZ190eXBlIGFnZzwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtcGNfdHlwZSBnYW1nPC9k
aXY+WzI0XVBFVFNDIEVSUk9SOiAtLS0tLS0tLS0tLS0tLS0tRW5kIG9mIEVycm9yIE1lc3NhZ2Ug
LS0tLS0tLXNlbmQgZW50aXJlIGVycm9yIG1lc3NhZ2UgdG8gcGV0c2MtbWFpbnRAbWNzLmFubC5n
b3YtLS0tLS0tLS0tPGJyPmBgYDxicj48YnI+QXR0YWNoZWQgd2l0aCB0aGlzIGVtYWlsIGlzIHRo
ZSBmdWxsIGVycm9yIGxvZyBhbmQgdGhlIHN1Ym1pdCBzY3JpcHQgZm9yIGEgCjgtbm9kZS82NC1H
UFUvNjQgTVBJIHJhbmsgam9iLiBJJ2xsIGFsc28gbm90ZSB0aGF0IHRoZSBzYW1lIHByb2dyYW0g
ZGlkIApub3QgY3Jhc2ggd2hlbiB1c2luZyBlaXRoZXIgMiBvciA0IG5vZGVzICg4IGFuZCAxNiBH
UFVzL01QSSByYW5rcykgYW5kIAphdHRhY2ggdGhvc2UgbG9ncyBhcyB3ZWxsIGlmIHRoYXQgaGVs
cHMuIENvdWxkIHNvbWVvbmUgbGV0IG1lIGtub3cgd2hhdCAKdGhpcyBlcnJvciBtZWFucyBhbmQg
d2hhdCBjYW4gYmUgZG9uZSB0byBwcmV2ZW50IGl0Pzxicj48YnI+PC9kaXY+PGRpdj48ZGl2IGlk
PSJTaWduYXR1cmUiPjxkaXY+Cgo8ZGl2PjwvZGl2Pgo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTpD
YWxpYnJpLEFyaWFsLEhlbHZldGljYSxzYW5zLXNlcmlmOyBmb250LXNpemU6MTJwdDsgY29sb3I6
cmdiKDAsMCwwKSI+ClRoYW5rIFlvdSw8YnI+CjxkaXYgZGlyPSJsdHIiPgo8ZGl2IGRpcj0ibHRy
Ij4KPGRpdj4KPGRpdiBkaXI9Imx0ciI+CjxkaXY+CjxkaXYgZGlyPSJsdHIiPgo8ZGl2IHN0eWxl
PSJmb250LXNpemU6MTIuOHB4Ij5TYWppZCBBbGkgKGhlL2hpbSkgfCBSZXNlYXJjaCBBc3NvY2lh
dGU8YnI+CjwvZGl2Pgo8ZGl2IHN0eWxlPSJmb250LXNpemU6MTIuOHB4Ij5TY2llbnRpZmljIENv
bXB1dGluZyBEaXZpc2lvbjxicj4KPC9kaXY+CjxkaXYgc3R5bGU9ImZvbnQtc2l6ZToxMi44cHgi
PkZlcm1pIE5hdGlvbmFsIEFjY2VsZXJhdG9yIExhYm9yYXRvcnk8YnI+CjwvZGl2Pgo8c3BhbiBz
dHlsZT0iZm9udC1zaXplOjEyLjhweCI+PGEgaHJlZj0iaHR0cDovL3Mtc2FqaWQtYWxpLmdpdGh1
Yi5pbyIgdGFyZ2V0PSJfYmxhbmsiPnMtc2FqaWQtYWxpLmdpdGh1Yi5pbzwvYT48L3NwYW4+PC9k
aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPGJyPgo8L2Rpdj4KCgo8L2Rp
dj48L2Rpdj48L2Rpdj4=" style="height:0px; width:0px; max-height:0px; max-width:0px; overflow:hidden; font-size:0em; padding:0px; margin:0px">
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>