<div dir="ltr">Also, try "-use_gpu_aware_mpi 0" to see if there is a difference.<div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 10, 2022 at 1:40 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Did it fail without GPU at 64 MPI ranks?</div><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 10, 2022 at 1:22 PM Sajid Ali Syed <<a href="mailto:sasyed@fnal.gov" target="_blank">sasyed@fnal.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">




<div dir="ltr">
<div>
<p style="margin:0px 0px 1.2em">Hi PETSc-developers, </p>
<p style="margin:0px 0px 1.2em">I’m seeing the following crash that occurs during the setup phase of the preconditioner when using multiple GPUs. The relevant error trace is shown below:
</p>
<pre style="font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block">(GTL DEBUG: 26) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272<br>[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: General MPI error
[24]PETSC ERROR: MPI error 1 Invalid buffer pointer
[24]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.
[24]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
...
[24]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[24]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[24]PETSC ERROR: #3 PetscSFBcastEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:218
[24]PETSC ERROR: #4 PetscSFBcastEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1499
[24]PETSC ERROR: #5 VecScatterEnd_Internal() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:87
[24]PETSC ERROR: #6 VecScatterEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:1366
[24]PETSC ERROR: #7 MatMult_MPIAIJCUSPARSE() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpicusparse/<a href="http://mpiaijcusparse.cu:302" target="_blank">mpiaijcusparse.cu:302</a>
[24]PETSC ERROR: #8 MatMult() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/interface/matrix.c:2438
[24]PETSC ERROR: #9 PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:730
[24]PETSC ERROR: #10 KSP_PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/petsc/private/kspimpl.h:421
[24]PETSC ERROR: #11 KSPGMRESCycle() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:162
[24]PETSC ERROR: #12 KSPSolve_GMRES() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:247
[24]PETSC ERROR: #13 KSPSolve_Private() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:925
[24]PETSC ERROR: #14 KSPSolve() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:1103
[24]PETSC ERROR: #15 PCGAMGOptProlongator_AGG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/agg.c:1127
[24]PETSC ERROR: #16 PCSetUp_GAMG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/gamg.c:626
[24]PETSC ERROR: #17 PCSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:1017
[24]PETSC ERROR: #18 KSPSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:417
[24]PETSC ERROR: #19 main() at poisson3d.c:69
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -dm_mat_type aijcusparse
[24]PETSC ERROR: -dm_vec_type cuda
[24]PETSC ERROR: -ksp_monitor
[24]PETSC ERROR: -ksp_norm_type unpreconditioned
[24]PETSC ERROR: -ksp_type cg
[24]PETSC ERROR: -ksp_view
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[24]PETSC ERROR: -mg_levels_ksp_type chebyshev
[24]PETSC ERROR: -mg_levels_pc_type jacobi
[24]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[24]PETSC ERROR: -pc_gamg_square_graph 1
[24]PETSC ERROR: -pc_gamg_threshold 0.0
[24]PETSC ERROR: -pc_gamg_threshold_scale 0.0
[24]PETSC ERROR: -pc_gamg_type agg
[24]PETSC ERROR: -pc_type gamg
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
</code></pre>
<p style="margin:0px 0px 1.2em">Attached with this email is the full error log and the submit script for a 8-node/64-GPU/64 MPI rank job. I’ll also note that the same program did not crash when using either 2 or 4 nodes (with 8 & 16 GPUs/MPI ranks
 respectively) and attach those logs as well if that helps. Could someone let me know what this error means and what can be done to prevent it?</p>
<p style="margin:0px 0px 1.2em">Thank You,<br>
Sajid Ali (he/him) | Research Associate</p>
<p style="margin:0px 0px 1.2em">Scientific Computing Division</p>
<p style="margin:0px 0px 1.2em">Fermi National Accelerator Laboratory</p>
<p style="margin:0px 0px 1.2em"><a href="http://s-sajid-ali.github.io" target="_blank">s-sajid-ali.github.io</a></p>
<div title="MDH:PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IENhbGlicmksIEFyaWFsLCBIZWx2ZXRpY2EsIHNhbnMt
c2VyaWY7IGZvbnQtc2l6ZTogMTJwdDsgY29sb3I6IHJnYigwLCAwLCAwKTsiPkhpIFBFVFNjLWRl
dmVsb3BlcnMsIDxicj48YnI+SSdtIHNlZWluZyB0aGUgZm9sbG93aW5nIGNyYXNoIHRoYXQgb2Nj
dXJzIGR1cmluZyB0aGUgc2V0dXAgcGhhc2Ugb2YgdGhlIHByZWNvbmRpdGlvbmVyIHdoZW4gdXNp
bmcgYSBtdWx0aXBsZSAoJmd0OzQpIG9mIEdQVXMuIFRoZSByZWxldmFudCBlcnJvciB0cmFjZSBp
cyBzaG93biBiZWxvdzogPGJyPjxicj5gYGA8YnI+KEdUTCBERUJVRzogMjYpIGN1SXBjT3Blbk1l
bUhhbmRsZTogcmVzb3VyY2UgYWxyZWFkeSBtYXBwZWQsIENVREFfRVJST1JfQUxSRUFEWV9NQVBQ
RUQsIGxpbmUgbm8gMjcyPGRpdj5bMjRdUEVUU0MgRVJST1I6IC0tLS0tLS0tLS0tLS0tLS0tLS0t
LSBFcnJvciBNZXNzYWdlIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IEdlbmVyYWwg
TVBJIGVycm9yPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IE1QSSBlcnJvciAxIEludmFsaWQg
YnVmZmVyIHBvaW50ZXI8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogU2VlIGh0dHBzOi8vcGV0
c2Mub3JnL3JlbGVhc2UvZmFxLyBmb3IgdHJvdWJsZSBzaG9vdGluZy48L2Rpdj5bMjRdUEVUU0Mg
RVJST1I6IFBldHNjIERldmVsb3BtZW50IEdJVCByZXZpc2lvbjogZjM1MWQ1NDk0YjU0NjJmNjJj
NDE5ZTAwNjQ1YWMyZTQ3N2I4OGNhZSAmbmJzcDtHSVQgRGF0ZTogMjAyMi0wMi0wOCAxNTowODox
OSArMDAwMDxicj4uLi48YnI+WzI0XVBFVFNDIEVSUk9SOiAjMSBQZXRzY1NGTGlua1dhaXRSZXF1
ZXN0c19NUEkoKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1h
aW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy92ZWMvaXMv
c2YvaW1wbHMvYmFzaWMvc2ZtcGkuYzo1NDxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMiBQZXRzY1NG
TGlua0ZpbmlzaENvbW11bmljYXRpb24oKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNr
LXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2st
c3JjL2luY2x1ZGUvLi4vc3JjL3ZlYy9pcy9zZi9pbXBscy9iYXNpYy9zZnBhY2suaDoyNzQ8L2Rp
dj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzMgUGV0c2NTRkJjYXN0RW5kX0Jhc2ljKCkgYXQgL3Rt
cC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJvM2Zp
cGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVjL2lzL3NmL2ltcGxzL2Jhc2ljL3Nm
YmFzaWMuYzoyMTg8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzQgUGV0c2NTRkJjYXN0RW5k
KCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2
a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVjL2lzL3NmL2ludGVy
ZmFjZS9zZi5jOjE0OTk8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzUgVmVjU2NhdHRlckVu
ZF9JbnRlcm5hbCgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3RhZ2UtcGV0c2Mt
bWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMvc3JjL3ZlYy9p
cy9zZi9pbnRlcmZhY2UvdnNjYXQuYzo4NzwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjNiBW
ZWNTY2F0dGVyRW5kKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRz
Yy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvdmVj
L2lzL3NmL2ludGVyZmFjZS92c2NhdC5jOjEzNjY8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjog
IzcgTWF0TXVsdF9NUElBSUpDVVNQQVJTRSgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3Bh
Y2stc3RhZ2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFj
ay1zcmMvc3JjL21hdC9pbXBscy9haWovbXBpL21waWN1c3BhcnNlL21waWFpamN1c3BhcnNlLmN1
OjMwMjwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjOCBNYXRNdWx0KCkgYXQgL3RtcC9zYWpp
ZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hl
eXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMvbWF0L2ludGVyZmFjZS9tYXRyaXguYzoyNDM4PC9k
aXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICM5IFBDQXBwbHlCQW9yQUIoKSBhdCAvdG1wL3Nhamlk
L3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5
dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9rc3AvcGMvaW50ZXJmYWNlL3ByZWNvbi5jOjczMDwv
ZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMTAgS1NQX1BDQXBwbHlCQW9yQUIoKSBhdCAvdG1w
L3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZrYmV4cm8zZmlw
ZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL2luY2x1ZGUvcGV0c2MvcHJpdmF0ZS9rc3BpbXBs
Lmg6NDIxPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxMSBLU1BHTVJFU0N5Y2xlKCkgYXQg
L3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWluLW1uajU2a2JleHJv
M2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3NwL2tzcC9pbXBscy9nbXJlcy9n
bXJlcy5jOjE2MjwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAjMTIgS1NQU29sdmVfR01SRVMo
KSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBldHNjLW1haW4tbW5qNTZr
YmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9rc3Ava3NwL2ltcGxzL2dt
cmVzL2dtcmVzLmM6MjQ3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxMyBLU1BTb2x2ZV9Q
cml2YXRlKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRzYy1tYWlu
LW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3NwL2tzcC9p
bnRlcmZhY2UvaXRmdW5jLmM6OTI1PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMxNCBLU1BT
b2x2ZSgpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3RhZ2UtcGV0c2MtbWFpbi1t
bmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMvc3JjL2tzcC9rc3AvaW50
ZXJmYWNlL2l0ZnVuYy5jOjExMDM8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogIzE1IFBDR0FN
R09wdFByb2xvbmdhdG9yX0FHRygpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3Rh
Z2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMv
c3JjL2tzcC9wYy9pbXBscy9nYW1nL2FnZy5jOjExMjc8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogIzE2IFBDU2V0VXBfR0FNRygpIGF0IC90bXAvc2FqaWQvc3BhY2stc3RhZ2Uvc3BhY2stc3Rh
Z2UtcGV0c2MtbWFpbi1tbmo1NmtiZXhybzNmaXBmNmtoZXl0dGxqendzczdmby9zcGFjay1zcmMv
c3JjL2tzcC9wYy9pbXBscy9nYW1nL2dhbWcuYzo2MjY8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogIzE3IFBDU2V0VXAoKSBhdCAvdG1wL3NhamlkL3NwYWNrLXN0YWdlL3NwYWNrLXN0YWdlLXBl
dHNjLW1haW4tbW5qNTZrYmV4cm8zZmlwZjZraGV5dHRsanp3c3M3Zm8vc3BhY2stc3JjL3NyYy9r
c3AvcGMvaW50ZXJmYWNlL3ByZWNvbi5jOjEwMTc8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjog
IzE4IEtTUFNldFVwKCkgYXQgL3RtcC9zYWppZC9zcGFjay1zdGFnZS9zcGFjay1zdGFnZS1wZXRz
Yy1tYWluLW1uajU2a2JleHJvM2ZpcGY2a2hleXR0bGp6d3NzN2ZvL3NwYWNrLXNyYy9zcmMva3Nw
L2tzcC9pbnRlcmZhY2UvaXRmdW5jLmM6NDE3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6ICMx
OSBtYWluKCkgYXQgcG9pc3NvbjNkLmM6Njk8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogUEVU
U2MgT3B0aW9uIFRhYmxlIGVudHJpZXM6PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IC1kbV9t
YXRfdHlwZSBhaWpjdXNwYXJzZTwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtZG1fdmVjX3R5
cGUgY3VkYTwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAta3NwX21vbml0b3I8L2Rpdj48ZGl2
PlsyNF1QRVRTQyBFUlJPUjogLWtzcF9ub3JtX3R5cGUgdW5wcmVjb25kaXRpb25lZDwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAta3NwX3R5cGUgY2c8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJP
UjogLWtzcF92aWV3PC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJST1I6IC1sb2dfdmlldzwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX2VzdGVpZ19rc3BfdHlwZSBjZzwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX2tzcF90eXBlIGNoZWJ5c2hldjwvZGl2Pjxk
aXY+WzI0XVBFVFNDIEVSUk9SOiAtbWdfbGV2ZWxzX3BjX3R5cGUgamFjb2JpPC9kaXY+PGRpdj5b
MjRdUEVUU0MgRVJST1I6IC1wY19nYW1nX2FnZ19uc21vb3RocyAxPC9kaXY+PGRpdj5bMjRdUEVU
U0MgRVJST1I6IC1wY19nYW1nX3NxdWFyZV9ncmFwaCAxPC9kaXY+PGRpdj5bMjRdUEVUU0MgRVJS
T1I6IC1wY19nYW1nX3RocmVzaG9sZCAwLjA8L2Rpdj48ZGl2PlsyNF1QRVRTQyBFUlJPUjogLXBj
X2dhbWdfdGhyZXNob2xkX3NjYWxlIDAuMDwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtcGNf
Z2FtZ190eXBlIGFnZzwvZGl2PjxkaXY+WzI0XVBFVFNDIEVSUk9SOiAtcGNfdHlwZSBnYW1nPC9k
aXY+WzI0XVBFVFNDIEVSUk9SOiAtLS0tLS0tLS0tLS0tLS0tRW5kIG9mIEVycm9yIE1lc3NhZ2Ug
LS0tLS0tLXNlbmQgZW50aXJlIGVycm9yIG1lc3NhZ2UgdG8gcGV0c2MtbWFpbnRAbWNzLmFubC5n
b3YtLS0tLS0tLS0tPGJyPmBgYDxicj48YnI+QXR0YWNoZWQgd2l0aCB0aGlzIGVtYWlsIGlzIHRo
ZSBmdWxsIGVycm9yIGxvZyBhbmQgdGhlIHN1Ym1pdCBzY3JpcHQgZm9yIGEgCjgtbm9kZS82NC1H
UFUvNjQgTVBJIHJhbmsgam9iLiBJJ2xsIGFsc28gbm90ZSB0aGF0IHRoZSBzYW1lIHByb2dyYW0g
ZGlkIApub3QgY3Jhc2ggd2hlbiB1c2luZyBlaXRoZXIgMiBvciA0IG5vZGVzICg4IGFuZCAxNiBH
UFVzL01QSSByYW5rcykgYW5kIAphdHRhY2ggdGhvc2UgbG9ncyBhcyB3ZWxsIGlmIHRoYXQgaGVs
cHMuIENvdWxkIHNvbWVvbmUgbGV0IG1lIGtub3cgd2hhdCAKdGhpcyBlcnJvciBtZWFucyBhbmQg
d2hhdCBjYW4gYmUgZG9uZSB0byBwcmV2ZW50IGl0Pzxicj48YnI+PC9kaXY+PGRpdj48ZGl2IGlk
PSJTaWduYXR1cmUiPjxkaXY+Cgo8ZGl2PjwvZGl2Pgo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTpD
YWxpYnJpLEFyaWFsLEhlbHZldGljYSxzYW5zLXNlcmlmOyBmb250LXNpemU6MTJwdDsgY29sb3I6
cmdiKDAsMCwwKSI+ClRoYW5rIFlvdSw8YnI+CjxkaXYgZGlyPSJsdHIiPgo8ZGl2IGRpcj0ibHRy
Ij4KPGRpdj4KPGRpdiBkaXI9Imx0ciI+CjxkaXY+CjxkaXYgZGlyPSJsdHIiPgo8ZGl2IHN0eWxl
PSJmb250LXNpemU6MTIuOHB4Ij5TYWppZCBBbGkgKGhlL2hpbSkgfCBSZXNlYXJjaCBBc3NvY2lh
dGU8YnI+CjwvZGl2Pgo8ZGl2IHN0eWxlPSJmb250LXNpemU6MTIuOHB4Ij5TY2llbnRpZmljIENv
bXB1dGluZyBEaXZpc2lvbjxicj4KPC9kaXY+CjxkaXYgc3R5bGU9ImZvbnQtc2l6ZToxMi44cHgi
PkZlcm1pIE5hdGlvbmFsIEFjY2VsZXJhdG9yIExhYm9yYXRvcnk8YnI+CjwvZGl2Pgo8c3BhbiBz
dHlsZT0iZm9udC1zaXplOjEyLjhweCI+PGEgaHJlZj0iaHR0cDovL3Mtc2FqaWQtYWxpLmdpdGh1
Yi5pbyIgdGFyZ2V0PSJfYmxhbmsiPnMtc2FqaWQtYWxpLmdpdGh1Yi5pbzwvYT48L3NwYW4+PC9k
aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPGJyPgo8L2Rpdj4KCgo8L2Rp
dj48L2Rpdj48L2Rpdj4=" style="height:0px;width:0px;max-height:0px;max-width:0px;overflow:hidden;font-size:0em;padding:0px;margin:0px">
​</div>
</div>
</div>

</blockquote></div>
</blockquote></div>